The Great Polish Opinion Poll Scandal!

Why did opinion pollsters get the result of the presidential election on Sunday so badly wrong?

If you work for a Polish opinion poll company, then I hope you are keeping your head down at the moment. Before you leave the house in the morning I hope you are disguising yourself with a wig and a pair of dark glasses. And if you have an opinion poll employee living next door to you, then you might be worried about them bringing down the price of property in your area.

And Polish pollsters should be feeling a little sheepish. Not only did all the main companies fail to predict the outcome and winner of Sunday’s presidential election, but they failed to predict accurately the result in the first round two weeks ago.

And in the lead up to the general election last month, most polls showed that the market orientated Civic Platform would be the largest party in parliament, with the conservative Law and Justice coming second. In the event, it was the other way round.

In the first round of the presidential election two weeks ago they said that the lead that Donald Tusk would take in to the second round would be around 6 percentage points. In the event it was only 3 percent.

And in the two weeks leading up to last Sunday’s two header, presidential shoot out, between Donald Tusk of Civic Platform and Lech Kaczynski of Law and Justice, virtually all polls predicted a win for Tusk, by a margin of six to ten percent.

And then, of course, Lech Kaczynski emerged as the eventual winner, by a margin of nine percent.

Cock up or cover up?

As you can see in the comments section in the last post on the beatroot, there have been people saying, “I told you so!’

Stefanmichnik comments after the result that pollsters dreaded came up:

“I've questioned you 2 weeks ago about opinion polls in Poland and the "special" role they play in the political campaigns in the postcommunist countries. What should the people who run Pentor, OBOP, Gfk and so one be doing this the morning?”

I imagine they should be putting a bullet in their heads, ‘stafan’.

As he points out, only one market research company picked up the swing to Kaczynski in those final days. And PGB pollsters has been consistently the only one to accurately reflect the amount of support for Kaczynski and his party right through this long election period.

So there has been a lot of hand ringing by market researchers in the wake of their darkest hour. Where did it all go so badly wrong?

But concern over the accuracy of political market research is not new here. Right wing groups have often pointed out how these companies underestimate their support. The leader of the far-right, League of Polish Families, Roman Giertych, points to the European parliamentary elections here last year, when pollsters predicted his party would get just 6% of the vote, when in fact, come polling day, they got 16 percent.

There has even been some dark mutterings from right wing circles that opinion poll companies are part of the post-communist establishment, and are, therefore, ‘part of Poland’s present day problems.’ Several of these companies, such as OBOP, CBOS and Demoskop have their roots back in the communist period, so you can see what the critics are insinuating.

In his post, stefanmichnik advises me to read a clipping from NIE magazine. This basically lists the technical problems and weaknesses of polls in Poland.

The head of the company that did best in the general and presidential elections, PGB, has said that one of things they don’t do, that some of the other companies who got it so badly wrong do, is use telephone polling. Telephone polling is usually the least accurate, and in Poland, especially so. Pollsters must get a representative sample of the population they are studying, and with only 3 out of 4 Poles having access to a fixed line phone, this is not the way to find that accurate sample. PGB also said that they never do surveys in people’s homes, but always on the street.

Concern over the inaccuracy of opinion polling has not been confined to Poland. Studies by Massachusetts University have found that those who are against liberalization of the economy in countries like Russia are regularly underrepresented in market research. Just like in Poland.

But there are no reasons to suppose that pollsters just ‘make things up out of thin air’, as some conspiracy theorists have done. Market researchers earn lots of profit every year from clients who presume that the results are accurate. They will not pay for things that are made up.

There are now calls for polling industry, worth billions of zlotys a year, to come under some sort of external control – maybe even from a government watchdog.

Just one thought to end with. A long time ago an American research institute went out into the street and asked people if they thought that the Metallic Metals Act should be passed through Congress. Forty percent of those polled said ‘Yes, it should.’ Of course, the Metallic Metals Act was a complete fiction.

When people are confronted by the nice lady from the opinion poll center, it seems that they can have opinions about things that simply don’t exist.

Perhaps people are telling pollsters, not what they think, but what they think the pollster wants them to think?