On Wednesday the 3rd, our too smart- too arrogant media will be asking why ‘the polls were wrong’. Let me approach the question honestly. I am a professional in the field. I’ve worked in the field for forty years and been an expert witness on survey research for over twenty years.
The theory of survey sampling is based on a mathematically perfect random selection from the universe of interest to the survey researcher. The mathematical proof of the validity of random sampling is simple and unquestionable.
In the case of polling, the universe to be sampled is all people who will vote in an election. Right off the bat you have a problem since this universe is roughly half of all people you could interview and the people you interview (a sample of the wrong universe) will not tell you the truth about whether they are even registered to vote. Got it?
Now add a few more problems. Only a tiny percent of the perfect sample are willing to be interviewed these days; maybe 5%. That really screws up the mathematics of sampling, making it irrelevant.
Recently, the use of cell phones has created two effects. One effect the media has noticed: about 6% of the population doesn’t have telephones with regular landlines. The media has not notice the second effect: everyone with a cell phone (more than half of the population) uses his or her landline differently these days. Most cell phone owners have a tape machine and they are much less likely to answer the phone live, than ever before.
Regardless of the radical change in the chance to get a reliable sample, survey researchers have always tried to correct for the changes with demographic correlates. (What you mean, Michael? ) If the final survey yields 40% women and 55% men and 5% refuse to answer, then the results are weighted so that the women get slightly more than a weight of 1 and the men get slightly less than a weight of 1. The same for other standard demographics, age, education and geographic location.
At this point, the weighting is a mess because people really lie about age and education. Worse, age and education are weaker and weaker correlates to voting habits than ever before. Urban older males are now the most volatile voting group and they are impossible to detect. Women and young people are shifting their allegiances.
Why do you know the polls are wrong? Because I looked at the 2002 Congressional polls and the 2003 California Recall polls and they were far off the final results. Far off in underestimating the volatile Independent vote. The only close poll in 2002 Congressional race was the CNN/USA/Gallup poll; it underestimated the Republican victory by much less than the other polls. So far in 2004, the CNN/USA/Gallop poll has been far to the Bush side by half a dozen points for this whole election run-up. Very different from the other widely oscillating polls. That is proof that the polls are wrong.
So what can we say about why the polls are wrong? Unfortunately we don’t have the material at hand to know what is specifically wrong. See the blog below to learn how we could get a good answer.