Saturday, October 6, 2012

Wisdom of the crowds revisited: 10,000 readers of the Volkskrant can't be wrong

We're about three weeks after the elections in The Netherlands but it seems like the discussion on the opinion polls is not finished yet. Especially in Belgium there is a lot of interest in the subject, probably because next week there will be municipal elections here.
The general feeling in the media is that the accuracy of the polls in The Netherlands was not good. I tried to put the so called bad results in perspective by expressing the difference between a poll and the actual result in function of the sample size. You can find that blog post here.
I would like to come back on this subject, after reading an article on Volkskrant.nl, the online version of De Volksksrant, a popular newspaper in The Netherlands.  If you speak Dutch you can read that article here. The title says that 9069 readers op Volkskrant.nl made a wrong prediction of the election results.

There are a few remarkable things about this article. To start with, the winner, i.e. the participant who deviated the least of the actual results, won with a deviation of 11 seats. I don't know what kind of predictions were allowed, but if you had to re-distribute the 150 available seats, the sum of the absolute value of the difference should always be an even number. I assume that the winner's guess did not sum to 150 or that smaller parties were grouped somehow.

A second remarkable point is that they admit that the average prediction of the participants deviated strongly from the actual results, just like the opinion polls. To start with, making an average prediction is probably not a good idea, furthermore the distance between that average and the actual results was 26 seats, which would put it (slightly) behind all other polls. Those other polls scored 24,24, 18 and 18 for "De Stemming", TNS, Maurice De Hond and Ipsos respectively. Apart from the fact that the average prediction of the participants of Volkskrant.nl who participated to this "game"  was actually worse than the worst polls, this admission contrasts sharply with the popular idea that opinion polls are either wrong, or when they are good, could as well have been predicted by an educated guess. Apparently this does not hold for the readers of the Volkskrant, or at least not those that participated. It also contrasts to what is known as "The wisdom of the crowds" which states that "the aggregation of information in groups results in decisions that are often better than could have been made by any single member of the group". It's very likely that James Surowiecki, the author of "The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations" was not referring to this type of election predictions, but it's nonetheless an interesting observation. 


A third remarkable point is that in a large group, i.e. 9096 participants, even the best result is still 11 seats away from the actual result. Without further information this is difficult to assess. So I made a small simulation study in which I randomly distributed the 150 available seats 9096 times and calculated the sum of the absolute values of the differences. Thankfully the results are very far off. In a second simulation, I attributed the 150 seats with a probability proportional to the election results of 2010. Notice though that the difference between the two election results is 46 seats, so it's not that hardly anything changed over those two elections. Either way, when I did this I got a result of 10 seats, even better than the Volkrant winner. But of course we're looking at the best result, the minimum of the sum of the absolute value of the differences, only, so this could have happened accidentally. So I replicated the experiment 100 times and found the following distribution.
 
The graph illustrates that the best Volkskrant result 11 (probably 10 or 12) is indeed not too bad from this perspective, but could also have been achieved by having 9096 readers randomly choosing with a probability proportional with the previous election results.
Some of you might be tempted to plot the differences observed with the opinion polls on this graph as well, but that would not be fair in that they did not result from 9096 replications.  However  if we would have had 9096 opinion polls and selected the best one, you could.