Site icon Polling Guru

Telephone or Online: Lessons from the Canadian Election Study

The latest and perhaps the most important study on the impact of survey mode was recently published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki in the Canadian Journal of Political Science using data from the 2015 Canadian Election Study (CES). The 2015 CES was a multi-mode study that uniquely included both a random telephone survey and an online survey using a commercial panel.

The use of a commercial panel is an important contribution considering the role of online panel surveys in Canada. The database of public polls created by Jenkins Research shows that 56% of all public survey releases (62% of all which were about public policy) were conducted online using a panel. While some of the panels used were recruited using random methods, it is striking how much online surveys are already the norm in Canada. Telephone surveys represent only 20% and IVR another 20%.

What did the authors find?

The analysis is comprehensive in that it considered how people answered the two different surveys (tendency to not answer or to go too quickly), how the different modes are associated with different response patterns (including questions that can be benchmarked against other reliable data), and how models predicting vote choice.

The authors conclude that “it is impossible to say which is more representative, which better tracks campaign dynamics, which provides better estimates of engagement, knowledge and sophistication among Canadian voters, which paints a clearer picture of political attitudes, and which allows a richer or more accurate understanding of voter choice.” [P.1032]

Implications for public polls

The bulk of publicly released polls in Canada are conducted online using panels so the lessons here speak directly to the confidence we can have in their accuracy at a time when many question poll methodologies. The results clearly identify differences between an RDD telephone survey and online surveys based on panels.

It is clear that at minimum we need to view surveys results with two things in mind: (1) that telephone surveys may over-represent the views of political engaged and trusting Canadians; (2) that the magnitude of social desirability bias in online surveys must be factored into understanding how Canadians answer questions about sensitive topics.

Academic citation: Breton, C., Cutler, F., Lachance, S., & Mierke-Zatwarnicki, A. (2017). Telephone versus Online Survey Modes for Election Studies: Comparing Canadian Public Opinion and Vote Choice in the 2015 Federal Election. Canadian Journal of Political Science, 4, 1–32.


Exit mobile version