Tuesday, October 25, 2005

The Great Polish Opinion Poll Scandal!

Why did opinion pollsters get the result of the presidential election on Sunday so badly wrong?

If you work for a Polish opinion poll company, then I hope you are keeping your head down at the moment. Before you leave the house in the morning I hope you are disguising yourself with a wig and a pair of dark glasses. And if you have an opinion poll employee living next door to you, then you might be worried about them bringing down the price of property in your area.

And Polish pollsters should be feeling a little sheepish. Not only did all the main companies fail to predict the outcome and winner of Sunday’s presidential election, but they failed to predict accurately the result in the first round two weeks ago.

And in the lead up to the general election last month, most polls showed that the market orientated Civic Platform would be the largest party in parliament, with the conservative Law and Justice coming second. In the event, it was the other way round.

In the first round of the presidential election two weeks ago they said that the lead that Donald Tusk would take in to the second round would be around 6 percentage points. In the event it was only 3 percent.

And in the two weeks leading up to last Sunday’s two header, presidential shoot out, between Donald Tusk of Civic Platform and Lech Kaczynski of Law and Justice, virtually all polls predicted a win for Tusk, by a margin of six to ten percent.

And then, of course, Lech Kaczynski emerged as the eventual winner, by a margin of nine percent.

Cock up or cover up?

As you can see in the comments section in the last post on the beatroot, there have been people saying, “I told you so!’

Stefanmichnik comments after the result that pollsters dreaded came up:

“I've questioned you 2 weeks ago about opinion polls in Poland and the "special" role they play in the political campaigns in the postcommunist countries. What should the people who run Pentor, OBOP, Gfk and so one be doing this the morning?”

I imagine they should be putting a bullet in their heads, ‘stafan’.

As he points out, only one market research company picked up the swing to Kaczynski in those final days. And PGB pollsters has been consistently the only one to accurately reflect the amount of support for Kaczynski and his party right through this long election period.

So there has been a lot of hand ringing by market researchers in the wake of their darkest hour. Where did it all go so badly wrong?

But concern over the accuracy of political market research is not new here. Right wing groups have often pointed out how these companies underestimate their support. The leader of the far-right, League of Polish Families, Roman Giertych, points to the European parliamentary elections here last year, when pollsters predicted his party would get just 6% of the vote, when in fact, come polling day, they got 16 percent.

There has even been some dark mutterings from right wing circles that opinion poll companies are part of the post-communist establishment, and are, therefore, ‘part of Poland’s present day problems.’ Several of these companies, such as OBOP, CBOS and Demoskop have their roots back in the communist period, so you can see what the critics are insinuating.

In his post, stefanmichnik advises me to read a clipping from NIE magazine. This basically lists the technical problems and weaknesses of polls in Poland.

The head of the company that did best in the general and presidential elections, PGB, has said that one of things they don’t do, that some of the other companies who got it so badly wrong do, is use telephone polling. Telephone polling is usually the least accurate, and in Poland, especially so. Pollsters must get a representative sample of the population they are studying, and with only 3 out of 4 Poles having access to a fixed line phone, this is not the way to find that accurate sample. PGB also said that they never do surveys in people’s homes, but always on the street.

Concern over the inaccuracy of opinion polling has not been confined to Poland. Studies by Massachusetts University have found that those who are against liberalization of the economy in countries like Russia are regularly underrepresented in market research. Just like in Poland.

But there are no reasons to suppose that pollsters just ‘make things up out of thin air’, as some conspiracy theorists have done. Market researchers earn lots of profit every year from clients who presume that the results are accurate. They will not pay for things that are made up.

There are now calls for polling industry, worth billions of zlotys a year, to come under some sort of external control – maybe even from a government watchdog.

Just one thought to end with. A long time ago an American research institute went out into the street and asked people if they thought that the Metallic Metals Act should be passed through Congress. Forty percent of those polled said ‘Yes, it should.’ Of course, the Metallic Metals Act was a complete fiction.

When people are confronted by the nice lady from the opinion poll center, it seems that they can have opinions about things that simply don’t exist.

Perhaps people are telling pollsters, not what they think, but what they think the pollster wants them to think?


michael farris said...

Doing phone polls in Poland is worse than useless and a sign that the pollsters are either stupid (if they're Polish and come up with a dumb idea like that) or don't know enough about the country they're working in (if they're foreign operations with Polish outlets - and Polish employees generally won't tell a foreign employer that they don't what they're doing).

Polish people's relations to phones over the last 50 or so years is an interesting story and almost all of it disqualifies that particular instrument (especially in its domestic incarnation) as a polling mechanism.

Gustav said...

Aww heck. PGB's last poll before the election had KaczyƄski at 50.2 and Tusk at 48.98 - a statistical tie if there ever was one.

How does that equal being accurate in an election with a nine-point spread!

Bialynia said...

I think the pollsters got it right, they simply got the turnout wrong. Zogby predicted a Kerry landslide on the eve of the election in 2004, he measured the turnout innaccurately. While he was the most accurate pollster in the 2000 election, it is difficult when you are basing your numbers on what the public says it will do, and not what it thinks.

I firmly believe that most people in Poland wanted Tusk to win, many slackers assumed he will and didn't vote (myself included in that group, I didn't vote at the Polish consulate even though I could have, so I get some blame). The religous rural voters came out in bigger bumbers and Kaczynski therefore won even though he had less support then Tusk.

michael farris said...

bialynia makes an important point, for most people upward mobility or any claim to intelligence in Poland more or less requires a slacker's attitude to politics and government.
this does not help the level of discourse (or the results of politics) at all (wild understatement).

beatroot said...

Can someone explain 'slackers attitude? Sorry, I'm a clueless Brit.

michael farris said...

A slacker is apathetic and/or an underachiever.

beatroot said...

But how, why, would someone who is upwardly mobile be an apathetic underacheiver?


michael farris said...

the and/or is important, in the context of upward mobility, apathy towards politics is de rigeur(sp?)

underachieving is important in the context of making it (or not) to the polling station. in W-wa this might mean actually cutting the weekend short.

beatroot said...

That's not a slacker...that's a lazy bastard!!!

Yeah, I met one of those on Monday...he was down about Kaczynski. When I asked him who he voted for he said, " Oh, I didn't vote...I was at my parents house in Poznan'!!!!!