In the first week of my graduate econometrics course, I try to emphasize the difference between using regression results for the purposes of prediction versus using them to make causal inferences. If your goal is to make predictions (without altering the data generating process), then simple regression, applied to observational data, can be a remarkably robust tool, and you don’t need to worry about designing a randomized controlled trial or coming up with a fancy quasi-experimental research design. This is also why polling is such a reliable predictive tool, and so I was surprised to see that in 2008 a two-day old CNN poll apparently did a terrible job of predicting the GOP Nevada caucuses results.
The linked reference, a tweet, claimed that CNN predicted vote shares of 29% for McCain, 19% for Romney, and 6% for Paul, but that the actual results came in at 51% for Romney, 14% for Paul, and 13% for McCain. The Nevada caucuses have very low turnout, so it’s conceivable that CNN completely mispredicted who would show up at the polls, but a deviation of that magnitude is still surprising. That the linked tweet had no reference piqued my suspicions, and a few minutes of searching on Lexis-Nexis confirmed them: the referenced CNN/ORC poll is a national level poll, not a Nevada state poll.
To eliminate any doubt, note the polling shares for each GOP candidate in the archived poll: McCain – 29%, Huckabee – 20%, Romney – 19%, Giuliani – 14%, and Paul – 6%. These numbers exactly match those in the tweet. The CNN/ORC poll was conducted on Jan 14-17, 2008, and the Nevada Republican caucuses occurred two days later on Jan 19, 2008. The probability that two separate CNN polls of different populations of voters, both supposedly conducted on Jan 17, 2008, would produce the exact same numbers for five candidates is on the order of 0.001 or less.
The bottom line:
- If a deviation looks too extreme to be plausible, maybe it is.
- Be even more suspicious of numbers that look implausible and have no references attached.