How not to read presidential polls: The increasing absurdity of media reports

By Abe Katsman

At first, it was just a trickle, a misguided throw-away line here and there, easily ignored. Then it started picking up momentum, showing up in one Israeli commentary after another. And now, it is conventional wisdom in the Israeli press and public that the U.S. election is already over, that polls show President Obama’s re-election is inevitable, and that Republican Mitt Romney might as well throw in the towel now.

Of course, this is nonsense. It is based on the most superficial reading of the most superficial polls. In fact, as described below, while poll results are all over the map, the most historically accurate pollsters consider the race tied.

This is hardly the first time the Israeli media herd has stampeded in the wrong direction based on flawed analysis. This time, everyone seems to be on board with the “only a miracle can save Romney” theme—this while U.S. economic numbers decline, the Middle East is on fire and al-Qaeda flags have been flown from area U.S. embassies, and we have yet to have even the first of the Presidential debates. (In fairness to the Israelis, it should be noted that much of the American media are, to varying degrees, on the same bandwagon, and are only now beginning to question to what degree the polls are skewed.)

But when even my own mathematician brother succumbs to this misreading of the race, it is time to explain what the polls do and do not say.

First, not all polls are created equal. There are all kinds of methodological differences among polling organizations in terms of targeted response rates, when voters are called (just how many Romney-voting small-business owners are home to answer a pollster’s weeknight call between 5:00 and 7:00 pm?), whether pollsters use live interviewers or computer calling, whether only landlines are called or also cellphones, how to weigh the non-response rate, and whether calls with no answer are called back. Statisticians argue endlessly about which methods should yield the most accurate results.

While various statistical methods may be sound, all polling analysis depends on the quality of assumptions and data inputted. Garbage in, garbage out: skewed data inputs lead to skewed poll results, no matter how brilliant any particular statistical methodology.

Decisions regarding those inputs are as much art as science.

And no assumption is as controversial or as influential on presidential polling results as “partisan weighting,” i.e., adjusting samples according to estimated party affiliation and turnout. The theory makes intuitive sense: if history and current trends indicate that, for example, 5% more of those voting are likely to be Republican, (“R+5”), the poll sample should be weighted accordingly, even if more Democrats are initially polled than Republicans.
How much weighting is appropriate? That is the crux of recent debates.

In the 2004 election in which George W. Bush defeated John Kerry, Democrats and Republicans turned out in roughly even numbers; but in 2008, the electorate consisted of 8% more Democrats than Republicans. Should a pollster adjust the results of his sample using the 2008 election as a model, which will strongly favor Obama, or the 2004 model, which will not?

The dirty secret of polling-as-news stories is that results are essentially pre-determined by the weighting model used. It should hardly come as news that a poll with 10% more Democrats sampled than Republicans will favor the Democratic candidate, or vice versa.

Even if we start at the 2008 D+8 figure, does anyone seriously believe that Democratic turnout will be the same for the reelection of a divisive incumbent in a stagnant economy with high unemployment as it was for the historical election of the first black President running on “hope and change” against a particularly weak candidate?

In fact, since 2008 the Democrats’ party identification advantage has not only eroded and disappeared, but has turned into a Republican advantage: the 2008 D+8 number identification from 2008 had already become R+1.3 by the 2010 midterm elections, which Obama himself described as “a shellacking” of the Democrats. And since then? Has anything in particular changed so dramatically as to move the needle in Obama’s direction? Apparently not, as the Republican advantage has only grown, and now stands at R+4.3%.

So, in an R+4.3 country, who in his right mind would take seriously a poll which skews current samples to D+8 or D+11, especially in states where that presumes a greater Democratic advantage than even 2008 would predict?

That is the question every news organization should be asking before reporting such polls. Yet, the Israeli press dutifully reports such poll results: just last week, it was big news in the Israeli media when The New York Times and CBS News published their own polls showing Obama up by 9 to 12 points in the important swing states of Ohio, Pennsylvania and Florida. Buried in the story, however, was the pro-Democrat weighting—as much as D+11. Is this really news—that a D+11 poll has Obama ahead by 11%?

In 2008, Obama won Florida by under 3%, and Ohio by under 5%. In other words, Florida and Ohio were less Democratic than the then-D+8 nation as a whole. As party affiliation has swung strongly in the Republican direction—and as Obama’s image has been badly tarnished in both state’s sizable Democrat-leaning Jewish communities—perhaps a bit of skepticism is in order before breathlessly reporting the New York Times/CBS numbers claiming double-digit leads for Obama.

Republicans consistently vote with their party at a greater rate than do Democrats. For 40 years, Republican candidates have won a remarkably consistent 84% (+/- 3%) of the votes of Republicans, while Democrats hover around 78% voter retention rates. Thus, under-weighting a Republican polling sample is apt to directly lower Republican poll results.

Another red flag is that many of the reported polls showing Obama way out front are polls of “adults” or “registered voters”; but the metric that matters and has always shown the most accuracy is that of “likely voters.” And those polls show a much tighter race.
Lastly, the polling organization rated most accurate since 2004 (Rasmussen) shows the race to currently be a statistical tie, at 47%-47%.

We have a long way to go before declaring this election over.

Abe Katsman is an American attorney living in Jerusalem. He serves as Counsel to Republicans Abroad Israel.

September 30, 2012 | 15 Comments »

Leave a Reply

15 Comments / 15 Comments

  2. @ Esther:
    Both Laura and Obama supporter better learn their history. If you look at Rasmussen it is 1980 all over again! You guys are betting on a losing horse.

  5. The media is trying to create a self-fulfilling prophesy by making the public believe it is all but over in favor of Obama.

    We no longer have a media that simply reports events, but instead creates stories to push forward a narrative favorable to their left wing agenda. NOTHING, and I mean, NOTHING they report on should be believed. NBC lies outright. Another MSNBC Scandal? Blaze Readers at Campaign Event Claim Network Misled in Video of Rally Chant (With Audio Comparison)

    Anyway, given Obama’s staggering failures and catastrophes, the polls shouldn’t be tied, Romney should be leading by double digits.

  7. @ Stanley J. revich:

    Living in the battleground state of Ohio, I can vouch for this. We aren’t hearing much about the Tea Party in the media, but I am connected with them, and I can assure you that they are VERY active in the ways in which Stan here described, especially his points #1 and #3 (i.e., poll watchers, GOP turnout).

    There are atill a lot of stupid people out there, however, and the fact that we’ve got the most corrupt and biased media in modern history supporting Obama is probably worth at least ten points for him. If we had the media of say, thirty years ago, Obama never would have been nominated, let alone elected. A minimaally honest media treatment of Obama today should, by all rights, have Romney ten to twenty points ahead.

    I expect Romney to win, but I am not as confident as I was in months past. Six months ago, I thought the only ways Obama could win involved the following scenarios, in descending order of likelihood:

    – Ron Paul third party run.

    – Obama – yes, Obama – hits Iran.

    – Election is rigged somehow (e.g., buggering the vote tallying computers, etc.).

    – “National emergency” declared that suspends elections indefinitely, leaving Obama as de facto dictator (what Ferdinand Marcos did in the Philippines in ’68).

    Now, with just over a month to go before elections, we can pretty much rule out my first two points.

    I still consider #4 highly unlikely. You need the backing of the military to make that stick, and the military brass despises Obama.

    That leaves #3. Like Stan, I don’t put that past Obama & Co. at all.

    We live in way too interesting times.

    But remember, everybody… Back in ’80, Carter was leading Reagan by as much as 8 points in the polls even during October. Reagan did not start to pull ahead until literally the week before the election, and he won by a much greater margin than predicted by ANY polls of that time.

    If polls really did predict the winner with absolute reliability, Kerry would have won in ’04. He outspent, Bush, too. And, the media also hated Bush, and that was pretty much the same media we have today.

    Romney should still win.

  8. Poor Esther, living in Never-Never Land. There is only ONE way that Obama can win — by cheating at the polls and fixing the results. Unfortunately I would not put this possibility past the Democrats. They will lie, cheat or steal, do anything, to win this election because for these ideological nitwits the only thing is winning.
    Has no one asked why the Tea Party has been so quiet out in the public realm? It is because they have been concentrating on three important avenues. One is training people as watchers at the voting stations; two, concentrating on the Congressional and State elections and, three, organizing to turn out the vote for Republicans. The election will be won or lost in these 3 areas. The Tea Party is not about to squander their money on useless politicking in a biased media.

  12. Its SO good there is someone who can see the truth about the media–and how they are lying to help Barack Obama get illegally reelected!

    It was just last week Ted Belman posted an article about Romney losing in the polls and claiming that Barack Obama is so likeable, and the American people want to pay more taxes–and this after everything since the 2008 and 2010 election–the 2010 election in which the American people showed there dislike of Obama policies!

    The truth is–Barack Obama is behind–and anyone who believes there will be a turnout in favor of him as there was in 2008–simply just doesn’t know how to think critically–the turnout for Barack Obama this time around will be much less in every significant group–even among the Jews.

  13. A landslide for Obama is the likely scenario. Romney has been a flop his pick of Ryan hurt him with seniors, and the American public is very comfortable with Obama. He’s a Democratic Reagan, his personal appeal matters more than his policies or performance.