Public Opinion Research Telephone or Online: Lessons from the Canadian Election Study

Telephone or Online: Lessons from the Canadian Election Study

Findings from recent study published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki

-

The latest and perhaps the most important study on the impact of survey mode was recently published by Breton, Cutler, Lachance, and Mierke-Zatwarnicki in the Canadian Journal of Political Science using data from the 2015 Canadian Election Study (CES). The 2015 CES was a multi-mode study that uniquely included both a random telephone survey and an online survey using a commercial panel.

The use of a commercial panel is an important contribution considering the role of online panel surveys in Canada. The database of public polls created by Jenkins Research shows that 56% of all public survey releases (62% of all which were about public policy) were conducted online using a panel. While some of the panels used were recruited using random methods, it is striking how much online surveys are already the norm in Canada. Telephone surveys represent only 20% and IVR another 20%.

What did the authors find?

The analysis is comprehensive in that it considered how people answered the two different surveys (tendency to not answer or to go too quickly), how the different modes are associated with different response patterns (including questions that can be benchmarked against other reliable data), and how models predicting vote choice.

  • The telephone and online samples did differ from each other and benchmarks in the following ways: over-representing 50+ year olds and atheists (both), over-representing the unemployed (online), and over-representing people who volunteer (telephone). The sample are “virtually indistinguishable on 2015 vote intention”[p.1015].
  • Both samples did a similar job capturing the campaign dynamics, though the telephone trend had a notable divergence in the middle of the campaign that lasted for a couple days.
  • On measures of political engagement the telephone results did differ from the online results. As the authors observed, “the differences are not enormous but phone respondents are significantly more interested in politics and reported more attention to the campaign” [p.1020].  Telephone respondents also reported paying more attention to news.
  • On political attitudes, some of the largest differences were observed. Many of which are consistent with social desirability bias.
    • Domain specific spending preferences were similar across the two modes except when it came to immigration. Similiarily, the differences across most policy questions were not significant.
    • Economic attitudes were significantly different with online results more negative.
    • Web respondents were less trusting and this is consistent with social trust being one factor in predicting people be willing to do the telephone survey.
    • Feelings about groups are notably different with online respondents more negative about groups such as politicians, Muslims, immigrants, feminists and gays.

The authors conclude that “it is impossible to say which is more representative, which better tracks campaign dynamics, which provides better estimates of engagement, knowledge and sophistication among Canadian voters, which paints a clearer picture of political attitudes, and which allows a richer or more accurate understanding of voter choice.” [P.1032]

Implications for public polls

The bulk of publicly released polls in Canada are conducted online using panels so the lessons here speak directly to the confidence we can have in their accuracy at a time when many question poll methodologies. The results clearly identify differences between an RDD telephone survey and online surveys based on panels.

It is clear that at minimum we need to view surveys results with two things in mind: (1) that telephone surveys may over-represent the views of political engaged and trusting Canadians; (2) that the magnitude of social desirability bias in online surveys must be factored into understanding how Canadians answer questions about sensitive topics.


Academic citation: Breton, C., Cutler, F., Lachance, S., & Mierke-Zatwarnicki, A. (2017). Telephone versus Online Survey Modes for Election Studies: Comparing Canadian Public Opinion and Vote Choice in the 2015 Federal Election. Canadian Journal of Political Science, 4, 1–32. http://doi.org/10.1017/S0008423917000610 https://www.cambridge.org/core/journals/canadian-journal-of-political-science-revue-canadienne-de-science-politique/article/telephone-versus-online-survey-modes-for-election-studies-comparing-canadian-public-opinion-and-vote-choice-in-the-2015-federal-election/B3F64CA440B8E927853DA71AD8BFFB69


 

Latest news

Social Progress versus Economic Growth in a Post-Pandemic World

Canadians are divided about what should be the post-pandemic focus. Half want to prioritize social progress (improved...

Canadians Approve Active Organ Donor Registration

Every year there is a shortage of organ donors in Canada. Canada requires people to choose to...

Assisted Dying: the Challenge of Aligning Policy with Public Opinion

In 2017, assisted dying, that is the ability to request that a physician provide help in ending...

Life after Isolation: How the Pandemic Might Change Everything or Nothing about the Post-COVID-19 Workplace

Predicting the future is always a task for the brave or the foolish. When history offers clear...

Is Concern with COVID-19 Cratering or Just Levelling off in Canada?

We have reached a pivot point in the COVID-19 impact in Canada. While Canadians were slow to...

COVID-19 Polls: Public Opinion in the Time of Pandemic

2020 will be the year of COVID-19 polls. As the Coronavirus spread throughout Canada, polling companies expanded...

Must read

Under Strain the Most Pressing Worry | Health Care System

Compared to the rest of the world, Canadians see...

A country of free-traders?

Trump's threat to cancel NAFTA and the series of...

You might also likeRELATED
Recommended to you