The latest and perhaps the most important study on the impact of survey mode was recently published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki in the Canadian Journal of Political Science using data from the 2015 Canadian Election Study (CES). The 2015 CES was a multi-mode study that uniquely included both a random telephone survey and an online survey using a commercial panel.
The use of a commercial panel is an important contribution considering the role of online panel surveys in Canada. The database of public polls created by Jenkins Research shows that 56% of all public survey releases (62% of all which were about public policy) were conducted online using a panel. While some of the panels used were recruited using random methods, it is striking how much online surveys are already the norm in Canada. Telephone surveys represent only 20% and IVR another 20%.
What did the authors find?
The analysis is comprehensive in that it considered how people answered the two different surveys (tendency to not answer or to go too quickly), how the different modes are associated with different response patterns (including questions that can be benchmarked against other reliable data), and how models predicting vote choice.
- The telephone and online samples did differ from each other and benchmarks in the following ways: over-representing 50+ year olds and atheists (both), over-representing the unemployed (online), and over-representing people who volunteer (telephone). The sample are “virtually indistinguishable on 2015 vote intention”[p.1015].
- Both samples did a similar job capturing the campaign dynamics, though the telephone trend had a notable divergence in the middle of the campaign that lasted for a couple days.
- On measures of political engagement the telephone results did differ from the online results. As the authors observed, “the differences are not enormous but phone respondents are significantly more interested in politics and reported more attention to the campaign” [p.1020]. Telephone respondents also reported paying more attention to news.
- On political attitudes, some of the largest differences were observed. Many of which are consistent with social desirability bias.
- Domain specific spending preferences were similar across the two modes except when it came to immigration. Similiarily, the differences across most policy questions were not significant.
- Economic attitudes were significantly different with online results more negative.
- Web respondents were less trusting and this is consistent with social trust being one factor in predicting people be willing to do the telephone survey.
- Feelings about groups are notably different with online respondents more negative about groups such as politicians, Muslims, immigrants, feminists and gays.
The authors conclude that “it is impossible to say which is more representative, which better tracks campaign dynamics, which provides better estimates of engagement, knowledge and sophistication among Canadian voters, which paints a clearer picture of political attitudes, and which allows a richer or more accurate understanding of voter choice.” [P.1032]
Implications for public polls
The bulk of publicly released polls in Canada are conducted online using panels so the lessons here speak directly to the confidence we can have in their accuracy at a time when many question poll methodologies. The results clearly identify differences between an RDD telephone survey and online surveys based on panels.
It is clear that at minimum we need to view surveys results with two things in mind: (1) that telephone surveys may over-represent the views of political engaged and trusting Canadians; (2) that the magnitude of social desirability bias in online surveys must be factored into understanding how Canadians answer questions about sensitive topics.
Academic citation: Breton, C., Cutler, F., Lachance, S., & Mierke-Zatwarnicki, A. (2017). Telephone versus Online Survey Modes for Election Studies: Comparing Canadian Public Opinion and Vote Choice in the 2015 Federal Election. Canadian Journal of Political Science, 4, 1–32. http://doi.org/10.1017/S0008423917000610 https://www.cambridge.org/core/journals/canadian-journal-of-political-science-revue-canadienne-de-science-politique/article/telephone-versus-online-survey-modes-for-election-studies-comparing-canadian-public-opinion-and-vote-choice-in-the-2015-federal-election/B3F64CA440B8E927853DA71AD8BFFB69