Demise of TV/Radio or Researchers Chasing a Story?

It seems that every week or two a new study shouts out on Twitter or social media declaring that the traditional ways of consuming TV and radio are being pushed aside and will soon be replaced by Wi-Fi TV, streaming smartphones, VOD, tablets, the second screen, etc.  Journalists faced with voracious editors lap this up and write about these sexy findings.   

Research results from the MTM (Media Technology Monitor) pop up in news stories on a regular basis. The MTM is conducted by two respected research firms in English and French Canada, marketed by BBM Analytics and accessed via a web portal designed and maintained by Forum Research. The funding is provided by a major broadcaster, although one has to have a keen eye to determine this.  It is not made clear who designs the study or writes the reports but we assume it is the sponsor.  One can purchase the results of MTM surveys at a steep discount, at a fraction of the original cost.  Or, you can visit the CRTC web site and get most of the MTM results for free or find most if not all the MTM trends in other sources, including Statistics Canada and BBM.  But if you want to know how many people in Alberta use game consoles versus an iPad to access Netflix, the MTM is for you. 

According to the MTM the majority of us are streaming audio, streaming video, watching Youtube regularly and presumably have little time left for anything else.  The MTM says in one place that 10% of us have Netflix, make that 12% in another place in the same report.  The MTM claims that 13% of Canadians subscribe to satellite radio but the sole satellite radio company says that its subscriber total is less than half that. However, the MTM didn't ask respondents if they personally had a Sirius/XM subscription and it is unclear if the MTM results are household or personal measures.  Respondents may have been confused and this may be why the MTM concludes that listening to podcasts is already on the decline, which seems counterintuitive. Perhaps that also explains why the CRTC published MTM  technology penetration statistics for VOD, etc. until 2011 and became more selective last year.

And therein is the biggest issue that surveys like the MTM represent. Just how valid are MTM-type surveys; do they over-estimate the adoption of new technologies and their impact on traditional media?  Are they so confusing to respondents that trends can't be relied on?
.

Let’s consider some research from CMRI’s annual surveys for comparative purposes.  CMRI has conducted research into the new technologies for the past 10 years; this includes VOD (Video-on-Demand) and PPV (Pay-Per-View).  CMRI’s Media Trends Survey, conducted by mail,  reveals that interest in and use of these new ways of consuming TV is not as important as the MTM suggests.  PPV, the oldest of these new ways of watching TV has only been used once a month or more by well under 10% of Anglophone Canadians (Francophones are similar) in ten consecutive surveys.  The vast majority of viewers say they rarely or never use PPV. It is not that the Media Trends Survey lacks for new technology—the percentage of respondents who have Netflix, smartphones, PVRs, etc. is in line with other industry estimates.

The same pattern is true of VOD.  In the past six surveys we have asked people about VOD and each year less than 10% of respondents indicated that they already use VOD and of those that do not, a large majority said that they were unlikely to do so.  As with PPV, there is no pattern of increased use.  

For the better part of the past century survey researchers have learned by trial and error and best practices have been documented in text books and journals.  In the 1995 Canadian Journal of Marketing Research I summarized much of what we had learned about surveys dealing with new media technologies (the "electronic highway" back then). Researchers face many of the same challenges today. 

Surveys that deal with new technology, such as the MTM, can over-estimate use of new media either because of sampling techniques which under-represent everyday persons (a common problem with online samples) or by structuring the interview and questionnaire in ways that encourage respondents to say they use the new technology. The MTM surveys appear to fall into the latter category.

The MTM survey is conducted by phone, is very detailed and would challenge many people, attempting to capture everything there is to know about old and new electronic media in one interview.  Research about new technologies is exceedingly difficult to do and is rarely done over the phone successfully. Phone surveys are excellent for testing response to and gauging atttiudes toward well-known products and services.  

The MTM questionnaire assumes that the respondent possesses a great deal of knowledge about new technologies, such as VOD, satellite radio, internet streaming, downloading, podcasting, etc.  Respondents are not given a description of VOD, HDTV, set-top boxes, podcasting, etc.  and many people are unfamiliar with such terms. Assuming knowledge about new technologies can make respondents feel poorly informed, not up-to-date, and, to meet the expectations of the interviewer, lead some to exaggerate how much they use new technologies.  The questionnaire also tips off that the broadcaster is the sponsor, which can create bias.   The current questionnaire is not available on the MTM web site but earlier versions were. 

The science in surveys consists of the sampling design and data analysis; the art in surveys lies in the interface with respondents and the design of the questionnaire. 

The 2011 survey results are from CMRI's Media Trends Survey conducted November-December 2011 among a representative national sample of approximately 900 Anglophone respondents aged 18-plus.  Margin of error +/-3.3%.  The Media Trends Survey has been conducted for ten consecutive years and has surveyed over 15,000 Canadians in total. In our analysis we usually only report Anglophone results.   Both Anglophones and Francophones have been surveyed in this period, using questionnaires in each respective language.  Francophones have been surveyed in 5 of the 10 years.  To compensate for poorer response rates among younger adults results are statistically weighted in keeping with industry standards.  It is the only survey to have measured media use and attitudes continuously over this decade. The Media Trends Survey is not sponsored by any one industry or affiliated with a media company.