Public Opinion Research Telephone or Online: Lessons from the Canadian Election Study

Telephone or Online: Lessons from the Canadian Election Study

Findings from recent study published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki

-

The latest and perhaps the most important study on the impact of survey mode was recently published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki in the Canadian Journal of Political Science using data from the 2015 Canadian Election Study (CES). The 2015 CES was a multi-mode study that uniquely included both a random telephone survey and an online survey using a commercial panel.

The use of a commercial panel is an important contribution considering the role of online panel surveys in Canada. The database of public polls created by Jenkins Research shows that 56% of all public survey releases (62% of all which were about public policy) were conducted online using a panel. While some of the panels used were recruited using random methods, it is striking how much online surveys are already the norm in Canada. Telephone surveys represent only 20% and IVR another 20%.

What did the authors find?

The analysis is comprehensive in that it considered how people answered the two different surveys (tendency to not answer or to go too quickly), how the different modes are associated with different response patterns (including questions that can be benchmarked against other reliable data), and how models predicting vote choice.

  • The telephone and online samples did differ from each other and benchmarks in the following ways: over-representing 50+ year olds and atheists (both), over-representing the unemployed (online), and over-representing people who volunteer (telephone). The sample are “virtually indistinguishable on 2015 vote intention”[p.1015].
  • Both samples did a similar job capturing the campaign dynamics, though the telephone trend had a notable divergence in the middle of the campaign that lasted for a couple days.
  • On measures of political engagement the telephone results did differ from the online results. As the authors observed, “the differences are not enormous but phone respondents are significantly more interested in politics and reported more attention to the campaign” [p.1020].  Telephone respondents also reported paying more attention to news.
  • On political attitudes, some of the largest differences were observed. Many of which are consistent with social desirability bias.
    • Domain specific spending preferences were similar across the two modes except when it came to immigration. Similiarily, the differences across most policy questions were not significant.
    • Economic attitudes were significantly different with online results more negative.
    • Web respondents were less trusting and this is consistent with social trust being one factor in predicting people be willing to do the telephone survey.
    • Feelings about groups are notably different with online respondents more negative about groups such as politicians, Muslims, immigrants, feminists and gays.

The authors conclude that “it is impossible to say which is more representative, which better tracks campaign dynamics, which provides better estimates of engagement, knowledge and sophistication among Canadian voters, which paints a clearer picture of political attitudes, and which allows a richer or more accurate understanding of voter choice.” [P.1032]

Implications for public polls

The bulk of publicly released polls in Canada are conducted online using panels so the lessons here speak directly to the confidence we can have in their accuracy at a time when many question poll methodologies. The results clearly identify differences between an RDD telephone survey and online surveys based on panels.

It is clear that at minimum we need to view surveys results with two things in mind: (1) that telephone surveys may over-represent the views of political engaged and trusting Canadians; (2) that the magnitude of social desirability bias in online surveys must be factored into understanding how Canadians answer questions about sensitive topics.


Academic citation: Breton, C., Cutler, F., Lachance, S., & Mierke-Zatwarnicki, A. (2017). Telephone versus Online Survey Modes for Election Studies: Comparing Canadian Public Opinion and Vote Choice in the 2015 Federal Election. Canadian Journal of Political Science, 4, 1–32. http://doi.org/10.1017/S0008423917000610 https://www.cambridge.org/core/journals/canadian-journal-of-political-science-revue-canadienne-de-science-politique/article/telephone-versus-online-survey-modes-for-election-studies-comparing-canadian-public-opinion-and-vote-choice-in-the-2015-federal-election/B3F64CA440B8E927853DA71AD8BFFB69


 

Latest news

Is Democracy Working? Satisfaction with Democracy in Canada

The 2019 federal election exacerbated regional tensions in Canada. While regional premiers were already opposed to key...

All the Polls Fit to Print: 2019 in the Polls

Public polls tell us a lot about what is on the minds of Canadians. Or at least...

Holiday Season | December

Each year it seems that the very question of the meaning of the holidays gets asked more...

Gig Economy: New Economy or More of the Same?

The 'gig' economy is not new. Informal work arrangements have always been part of the economy. What...

Christmas is Too Commercial

It is that time of year. With pre-Christmas sales already well underway, a number of polls reflect...

Libertarianism: How Canadians in the 1980s Saw Personal Autonomy

The 1980s are an interesting decade. Baby Boomers entered the workforce, got married and had kids. The...

Must read

Arming Saudi Arabia: The Collision of Values and Business

A look back at four polls which have addressed the export of arms to Saudi Arabia with a focus on the inherent challenge of reconciling the public's desire for an ethical policy with the realities of arms exports.

You might also likeRELATED
Recommended to you