Polls: The good, the bad and the ugly

TOM REGAN: SUMMONING ORENDA
June 4, 2016

“… dogs know best what to do with polls.” — John Diefenbaker, the late former prime minister of Canada

It probably sounded better when the late Canadian Prime Minister John Diefenbaker said it. That way you get the joke on ‘pole’ and ‘poll.’ No matter, his contempt for polls was legendary. In the late 60s and early 70s, he felt that being told about the results of a poll was often just as bad as being told an outright lie. He was probably not wrong.

Back then there was a handful of organizations polling the public. Now there seem to be more polls than flowers in spring. Every day some business, university or news organization announces the results of some poll or survey. This is particularly bad in a country like America, with its never-ending election campaigns.1

The funny thing is, with all this constant polling you would think polling would be much better. It’s not. Aside from a group of about 8-10 credible pollsters, most polls are  ‘meh.’ Perhaps not “outright lies,” but not worth much more than what you might pay for stock in Lehman Brothers these days. To make matters worse, these relatively worthless polls are reported on by a breathless media, hungry for polls that suit the narrative they want to sell to the public.

So here are a few suggestions (based on my personal experience doing polls in Canada and the US, and the wise words of the master of polling Nate Silver) about what to watch out for in polls, how you can tell a good one from a bad one, and why you never, ever, ever bet your house on one poll only.

There are two big problems pollsters face: 1) fewer and fewer people have landline phones – in particular younger people and minorities – which makes it harder and harder to do live interviews, and, 2) people are fed up being polled (thanks to telemarketing) and so either never answer the phone, or just hang up. So one way to tell a good poll is if the pollster conducted live interviews (and not robocalls) to both landlines and cellphones. Also, check the poll’s internal data to see its demographic makeup.  A poll of only younger white males will not produce an accurate picture of the raec between Hillary Clinton and Bernie Sanders, for instance.

Pay much more attention to a poll of likely voters than to a poll of registered voters. There are lots of registered voters in the US. Not all of them will actually vote in the election. (One poll included data from people who weren’t registered at all.) Likely voters have had a record of past voting in a variety of elections. When they tell you they will be voting, they mostly mean it. A poll of likely voters will give you a much more accurate picture than one of registered voters.

Go to a site like fivethirtyeight.com (my favourite), HuffPost Pollster, or RealClearPolitics to see how the entity that did the poll has performed in the past. Some pollsters, like Rasmussen – one of the very worst in my opinion – has a strong GOP bias (according to 538.com’s ratings of pollsters) of almost 2% which means every poll that shows Clinton leading trump by only one percentage point is probably in reality closer to 3 or 4 percent, which most polls show. There are polls with strong Democratic biases as well, and they need to be regarded with just as much suspicion.

Also, one poll cannot be trusted, even if it makes your candidate or a position you support look great, because it may be an outlier. Look at the average of polls over the past month for a  better picture of the true nature of the race. Also, polls by good pollsters done just before an election can give a more accurate forecast of the results, but not always.

One of the reasons I love 538.com is that they not only do a forecast that looks at what the polls are saying, but they also do what they all a ‘polls-plus” forecast. This forecast is based on the Bayesian theory idea that you update your forecast about an election as more information becomes available. You don’t just stand put. Different factors affect how people vote and give a better hint than just polls. Some of these factors named by 538.com are “endorsements, state and national fundraising totals, favorability ratings, ideology ratings and national polls.” All information that helps you better understand where a race really stands. It is like taking into account stats but also the injury report, status of contract negotiations, and personal issues of your favorite ball player when guessing how he’ll do in the coming season.

But much of the media seems to understand none of this, or doesn’t want to. So I warn once again, never believe anything about a poll report, from particularly cable news. Cable news is not interested in the facts – it is interested in the horse race, and in raising money. Let me give you an example.

Earlier this month, there were four polls in California, two showing Hillary Clinton winning by only two points. The other two showed her winning by a margin of 18 points and 13 points. Both results are possible, depending on how the very large Hispanic population votes (see article below). But much of the media almost uniformly ignored the larger results. It’s not because they are pro-Sanders. It’s because they are pro-horse race. The two-point advantage fit the narrative they wanted to spin, so that’s what they sold. Only one journalist I could find, Paul Krugman of the New York Times, mentioned the 18-point poll, probably because he is a Clinton supporter (as am I).

The best journalism would be to report that either outcome is possible. But good journalism is both too wishy-washy and off-message for much of the media. So the horse race wins out.

So if you want to really get a good sense of what the polls are telling us, please go to fivethirtyeight.com. Nate Silver and his crew are the best. And if you get a chance, read his book, “The signal and the Noise” if you’re in a really geeky mood and want to understand how this stuff applies across a variety of fields. Because polls are a part of our lives and the more you understand the way they work, the better you’ll be informed.

Copyright Tom Regan 2016

Contact Tom Regan:  motnager@gmail.com

Notes:

  1. Constant US electioneering is the result of two factors: 1) politicians constantly looking for new sources of money and 2) TV and cable news networks trying to keep the money flowing in – so it is to the benefit of both of these groups that they just constantly promote the next time we need to visit the voting booth.

LINKS

Hispanic Voters Will Decide Bernie Sanders’s Fate in California, 538: http://fivethirtyeight.com/features/hispanic-voters-will-decide-bernie-sanderss-fate-in-california/

Fivethirtyeight.com Pollster Rankings: http://projects.fivethirtyeight.com/pollster-ratings/

 

 

Tom Regan Tom Regan has worked for the Canadian Broadcasting Corporation and with the National Film Board in Canada, and in the United States for the Christian Science Monitor, Boston Globe, and National Public Radio. A former executive director of the Online News Association in the U.S., he was a Nieman fellow at Harvard in 1991-92. He is based near Washington, D.C.

Return to Tom Regan’s page 

 

 

 

 ~~~

Facts and Opinions is a boutique journal of reporting and analysis in words and images, without borders. Independent, non-partisan and employee-owned, F&O is funded only by you, our readers. We are ad-free and spam-free, and do not solicit donations from partisan organizations. To continue we need a minimum payment of .27 for one story, or a sustaining donation. Visit our Subscribe page for details, or donate below. With enough supporters each paying a small amount, we will continue, and increase our original works like this.

F&O’s CONTENTS page is updated each Saturday. Sign up for emailed announcements of new work on our free FRONTLINES blog; find evidence-based reporting in Reports; commentary, analysis and creative non-fiction in OPINION-FEATURES; and image galleries in PHOTO-ESSAYS. If you value journalism please support F&O, and tell others about us.