How to interpret the polls in 2020 given what happened in 2016
OK, it's time. Let's do this thing.
Early in an election cycle, I will advise you to disregard the polls because voters are not sufficiently engaged in the election for the polls to be informative. Over time, they become more informative. There is no fixed date at which they become informative. It is not as though the pre-convention polls are irrelevant, and the post-convention polls are mountaintop truth, but as election day approaches, the information we gain from looking at the polls becomes gradually more informative. As of today, September 12, the election is close enough that we can begin to take the polls seriously. As I type that, though, we need to address the elephant/gorilla chimera in the room, illegally constructed in a bioengineering lab by someone who read too many H.G. Wells books. Could be worse, right? Could be something squamous and eldritch, and thanks to tv, I guess more people get that reference. Whatever. (And in case you were wondering, no I haven't seen it, so I have no opinion of it.)
Anyway, that chimerical beastie in the room, when it comes to polls, is 2016. As I regularly have to remind those who can't distinguish between two different forms of mathematical political science, the forecasting models got 2016 right, but the polls got 2016 wrong. So let's dig into the issues with polling, what happened in 2016, and what that means for interpreting the polls today.
If you head over to RealClearPolitics, they have Biden up nationwide by an average of 7.5 points as of this morning. Of course, that's merely a shorthand, and the best thing ever written on the "nationwide popular vote" is actually this Onion article. It isn't just that the "popular vote" is irrelevant. It doesn't even exist. Talking about it is like talking about the point totals on a chess board. After a checkmate. Stop it. Stop. It. That's not how we run this game, and since that's not how candidates strategize their campaigns, we don't know what the vote totals would be under the hypothetical counterfactual of an election under a nationwide popular vote, so whatever you are calling "the popular vote" isn't even that. It's nothing. It is illusory. Stop. It.
Moving on. We can look at the state-by-state totals. Right now, RCP has Biden just edging out Trump by an average of 1.2 points in Florida. Contrast that with Wisconsin, where RCP has Biden up by an average of 6.5 points. I'm done typing this out for you. It's time for you to start watching these numbers, which could change by the time you read this post anyway. It doesn't actually matter which polling aggregation method you use. RCP just goes with simple arithmetic means. Nate Silver adds what I call "statistical guitar face," but he's basically full of shit because what he adds beyond the arithmetic means misses the point.
So what's going on, according to the polls, right now? Basically, Biden is in the lead. Right about now, your response is... so was Clinton in 2016!
And that's why we need to deal with this. Consider two hypothetical polls, versus actual state totals.
Case 1: The poll result has the Democrat polling at 53%, and on election day, the Democrat wins 51%.
Case 2: The poll result has the Democrat polling at 51%, and on election day, the Democrat wins 49%.
The simple-minded response to Case 1 is to say that the poll got Case 1 "right" because it "predicted" that the Democrat would win, and the Democrat won. That same simple-minded approach to Case 2 would say that the poll got Case 2 "wrong" because it "predicted" that the Democrat would win, and the Democrat lost.
Do you see the problem here? In both cases, the poll overestimated the Democrat's vote share by 2 percentage points. And yet, we call it a "correct" assessment in Case 1, and an "incorrect" assessment in Case 2? Does this really make sense?
The issue is that when we are putting an estimate on the proportion of the vote a candidate will receive, then by definition, we aren't just making a 1 or 0 call. You may care only about 1 or 0, but that's not what we're doing. So let's elaborate further.
American politics are not, generally speaking, structured around class identity. Religion, however, plays a strong role in many aspects of American politics. Not so much sectarianism, but what we call "religiosity," despite the horribleness of the word. We measure it, often, by asking survey respondents questions like, how often do you attend religious services?
In the 2016 American National Election Studies survey, 30.6% claimed to attend religious services every week, and another 20.3% claimed to attend nearly every week (weighted). Um... sure. Yeah. Right.
The old joke we used to tell was that you could drive by a church on a Sunday morning and count the cars to compare to these numbers but right now that's... kind of bad methodology. Still, y'all know that these responses are called "lies," right? In technical, social science terminology, though, we have a fancy term for this kind of lie: a socially desirable response. People lie, and say, yeah, sure, I go to church every week, because they think that this particular survey answer makes them look, like, all "virtuous," 'n stuff.
If respondents lie in response to this question, does that mean we throw out responses to this question?
No!
Are the 30.6% who claim to attend religious services every week meaningfully different from the 28.1% who claim to attend only a few times a year (read: never)? Yes. Measurably. In oh, so many ways.
What's happening here is that peoples' responses are "biased," in the technical meaning of the term. People overstate their attendance at religious services because this country is filled with lying, posturing phonies who think they are supposed to pretend they are all "godly," 'n stuff, even though they are about as biblically literate as they are mathematically literate. (Or, I suppose, "literate.")
That bias is manifested when any individual over-reports service attendance. Yet, if each individual over-reports service attendance by a consistent amount, then differences between individuals are preserved, and in social scientific terms, those are usually what matter to us. If I want to know the effect, for example, of religiosity on various policy positions, then I can compare the positions of those who say they attend a few times a year, to those who claim to attend every week, and I get what I need, scientifically speaking. It is irrelevant to my question whether or not everyone is overstating their service attendance.
But of course, that's not quite what we're doing with an election poll. We care about the thing for which we have a potentially biased measure. If that's what's going on. So...
There has been long-running speculation in polling that some day, we'd see a "Bradley effect," or, "Wilder effect," in honor of Tom Bradley or Douglas Wilder, based on the notion that polls will overstate voters' willingness to vote for African-American candidates. The evidence for a Bradley effect, or any kind of systematic basis, is not strong. The irony, of course, is if we had a kind of reverse-Bradley effect for Trump. There are a variety of reasons we could see a reverse-Bradley effect for Trump.
So let's say there is a reverse-Bradley effect. Let's say polls understate Trump's support by 3 points or so. That would mean, what? It would mean that Biden is not leading in Florida, but that he is leading in Wisconsin. Understanding bias means that we don't throw out the surveys. We account for the bias in our interpretation. There are strong reasons to suspect that people will underreport their willingness to vote for Donald Trump. Knowing that, as we now do, we can factor that into the analysis.
The pollsters themselves are trying to do this. They are attempting to build likely voter screens and sample weights to figure out what the electorate will look like and compensate for bias to the degree that they can, but lying is a difficult thing for any methodology to address. Liars are the absolute worst, and I hate liars with every fiber of my being.
In this case, though, we simply adjust our interpretation of the world to account for the expectation of liar-ism.
So we're done now, right?
Not so fast. There is one more big point about 2016, and about 2020. This one doesn't get quite enough attention in the discussion of what happened in the 2016 polls. Strangely, though, it is kind of the big news from yesterday. Nora Dannehy, one of Durham's people, resigned in protest yesterday, reportedly because Bill Barr is-- surprise, surprise-- interfering with Durham's "investigation" into the Mueller investigation. Why? Barr wants to make sure Trump gets his October surprise out of Durham, and Dannehy was having none of it.
The thing is, there's a DoJ policy of not pulling any of this crap within a couple of months of an election. Bill Barr is absolutely dead-set on violating that policy. Why? Because that's how Trump won last time! Two weeks before the 2016 election, James Comey violated that policy, in the face of warnings from the rest of the DoJ, in order to announce a "re-opened investigation" into the Hillary Clinton email nonsense. Why? Someone found a laptop sitting around at the home of Anthony Weiner and Houma Abedin. All of the emails had already been checked, there was nothing there, Comey knew that, the re-opened investigation was bullshit, and as soon as Comey made the announcement, the polls moved "bigly." It was the only October surprise on record that actually moved the polls, and it did so by more than enough to swing the electoral college.
This... this is what Barr is planning. Well, not this precisely, but something like this. Barr is going to use the DoJ in a similar way to try to get late movement in the polls. Absolute, 100% guarantee.
And this matters for polling, for two reasons. First, the basic observation here about social science is that we can calculate "margins of error," which are based on sampling error. Get a sample of 700 or so, pretend it's random, and then because you could have gotten a weird sample, we estimate a 95% confidence interval such that we are 95% confident that an actual parameter falls within that range. But that's just sampling error. What's harder? Dealing with exogenous events. Big stuff that can come along and alter the landscape in some more fundamental way.
Coronavirus did that. We aren't likely to see anything that big, but stuff happens. In 2016, Comey happened.
And more complicatedly, he happened so close to the election that we didn't have very much post-Comey polling. Those "wrong" polls from 2016? Mostly pre-Comey. We just didn't have the time to do as much post-Comey polling as we needed to do. And there was volatility. Clinton's numbers dropped, then started to rebound, at the national level. State-level polling? We just couldn't do all of the state-level polling we needed to do. Point being, Comey didn't just screw up the entire 2016 election (which... he did). He screwed up our ability to poll the election.
To be sure, I just told you Barr is going to try the same thing. Of course, that may be tempered by our advanced knowledge that Barr will try it. Barr is about as subtle as an air raid klaxon.
So what does this mean? This, honestly, is harder. Once we know about a bias, we can account for it in our interpretation and analysis. An exogenous event, too late for polling to study? That's hard. Although technically, election interference from Barr will be an endogenous event rather than an exogenous event, for anyone being persnickety about econo-jargon.
Regardless, my general point remains that it is not as simple as saying that the polls were "wrong" in 2016, so ignore them. They were "biased," in 2016, but we can account for bias, knowing the bias. We must also be aware of the eventuality of an "October surprise," and be concerned about what happens if such an event occurs too late for us to poll for it.
Of course, none of this even begins to get at the difficulty of polling for an election that will be conducted largely via absentee balloting, nor the even more fundamental problem that we have a sitting president who has effectively announced his unwillingness to step down, preemptively declaring the entire process fraudulent.
So there's that. Yay?
Early in an election cycle, I will advise you to disregard the polls because voters are not sufficiently engaged in the election for the polls to be informative. Over time, they become more informative. There is no fixed date at which they become informative. It is not as though the pre-convention polls are irrelevant, and the post-convention polls are mountaintop truth, but as election day approaches, the information we gain from looking at the polls becomes gradually more informative. As of today, September 12, the election is close enough that we can begin to take the polls seriously. As I type that, though, we need to address the elephant/gorilla chimera in the room, illegally constructed in a bioengineering lab by someone who read too many H.G. Wells books. Could be worse, right? Could be something squamous and eldritch, and thanks to tv, I guess more people get that reference. Whatever. (And in case you were wondering, no I haven't seen it, so I have no opinion of it.)
Anyway, that chimerical beastie in the room, when it comes to polls, is 2016. As I regularly have to remind those who can't distinguish between two different forms of mathematical political science, the forecasting models got 2016 right, but the polls got 2016 wrong. So let's dig into the issues with polling, what happened in 2016, and what that means for interpreting the polls today.
If you head over to RealClearPolitics, they have Biden up nationwide by an average of 7.5 points as of this morning. Of course, that's merely a shorthand, and the best thing ever written on the "nationwide popular vote" is actually this Onion article. It isn't just that the "popular vote" is irrelevant. It doesn't even exist. Talking about it is like talking about the point totals on a chess board. After a checkmate. Stop it. Stop. It. That's not how we run this game, and since that's not how candidates strategize their campaigns, we don't know what the vote totals would be under the hypothetical counterfactual of an election under a nationwide popular vote, so whatever you are calling "the popular vote" isn't even that. It's nothing. It is illusory. Stop. It.
Moving on. We can look at the state-by-state totals. Right now, RCP has Biden just edging out Trump by an average of 1.2 points in Florida. Contrast that with Wisconsin, where RCP has Biden up by an average of 6.5 points. I'm done typing this out for you. It's time for you to start watching these numbers, which could change by the time you read this post anyway. It doesn't actually matter which polling aggregation method you use. RCP just goes with simple arithmetic means. Nate Silver adds what I call "statistical guitar face," but he's basically full of shit because what he adds beyond the arithmetic means misses the point.
So what's going on, according to the polls, right now? Basically, Biden is in the lead. Right about now, your response is... so was Clinton in 2016!
And that's why we need to deal with this. Consider two hypothetical polls, versus actual state totals.
Case 1: The poll result has the Democrat polling at 53%, and on election day, the Democrat wins 51%.
Case 2: The poll result has the Democrat polling at 51%, and on election day, the Democrat wins 49%.
The simple-minded response to Case 1 is to say that the poll got Case 1 "right" because it "predicted" that the Democrat would win, and the Democrat won. That same simple-minded approach to Case 2 would say that the poll got Case 2 "wrong" because it "predicted" that the Democrat would win, and the Democrat lost.
Do you see the problem here? In both cases, the poll overestimated the Democrat's vote share by 2 percentage points. And yet, we call it a "correct" assessment in Case 1, and an "incorrect" assessment in Case 2? Does this really make sense?
The issue is that when we are putting an estimate on the proportion of the vote a candidate will receive, then by definition, we aren't just making a 1 or 0 call. You may care only about 1 or 0, but that's not what we're doing. So let's elaborate further.
American politics are not, generally speaking, structured around class identity. Religion, however, plays a strong role in many aspects of American politics. Not so much sectarianism, but what we call "religiosity," despite the horribleness of the word. We measure it, often, by asking survey respondents questions like, how often do you attend religious services?
In the 2016 American National Election Studies survey, 30.6% claimed to attend religious services every week, and another 20.3% claimed to attend nearly every week (weighted). Um... sure. Yeah. Right.
The old joke we used to tell was that you could drive by a church on a Sunday morning and count the cars to compare to these numbers but right now that's... kind of bad methodology. Still, y'all know that these responses are called "lies," right? In technical, social science terminology, though, we have a fancy term for this kind of lie: a socially desirable response. People lie, and say, yeah, sure, I go to church every week, because they think that this particular survey answer makes them look, like, all "virtuous," 'n stuff.
If respondents lie in response to this question, does that mean we throw out responses to this question?
No!
Are the 30.6% who claim to attend religious services every week meaningfully different from the 28.1% who claim to attend only a few times a year (read: never)? Yes. Measurably. In oh, so many ways.
What's happening here is that peoples' responses are "biased," in the technical meaning of the term. People overstate their attendance at religious services because this country is filled with lying, posturing phonies who think they are supposed to pretend they are all "godly," 'n stuff, even though they are about as biblically literate as they are mathematically literate. (Or, I suppose, "literate.")
That bias is manifested when any individual over-reports service attendance. Yet, if each individual over-reports service attendance by a consistent amount, then differences between individuals are preserved, and in social scientific terms, those are usually what matter to us. If I want to know the effect, for example, of religiosity on various policy positions, then I can compare the positions of those who say they attend a few times a year, to those who claim to attend every week, and I get what I need, scientifically speaking. It is irrelevant to my question whether or not everyone is overstating their service attendance.
But of course, that's not quite what we're doing with an election poll. We care about the thing for which we have a potentially biased measure. If that's what's going on. So...
There has been long-running speculation in polling that some day, we'd see a "Bradley effect," or, "Wilder effect," in honor of Tom Bradley or Douglas Wilder, based on the notion that polls will overstate voters' willingness to vote for African-American candidates. The evidence for a Bradley effect, or any kind of systematic basis, is not strong. The irony, of course, is if we had a kind of reverse-Bradley effect for Trump. There are a variety of reasons we could see a reverse-Bradley effect for Trump.
So let's say there is a reverse-Bradley effect. Let's say polls understate Trump's support by 3 points or so. That would mean, what? It would mean that Biden is not leading in Florida, but that he is leading in Wisconsin. Understanding bias means that we don't throw out the surveys. We account for the bias in our interpretation. There are strong reasons to suspect that people will underreport their willingness to vote for Donald Trump. Knowing that, as we now do, we can factor that into the analysis.
The pollsters themselves are trying to do this. They are attempting to build likely voter screens and sample weights to figure out what the electorate will look like and compensate for bias to the degree that they can, but lying is a difficult thing for any methodology to address. Liars are the absolute worst, and I hate liars with every fiber of my being.
In this case, though, we simply adjust our interpretation of the world to account for the expectation of liar-ism.
So we're done now, right?
Not so fast. There is one more big point about 2016, and about 2020. This one doesn't get quite enough attention in the discussion of what happened in the 2016 polls. Strangely, though, it is kind of the big news from yesterday. Nora Dannehy, one of Durham's people, resigned in protest yesterday, reportedly because Bill Barr is-- surprise, surprise-- interfering with Durham's "investigation" into the Mueller investigation. Why? Barr wants to make sure Trump gets his October surprise out of Durham, and Dannehy was having none of it.
The thing is, there's a DoJ policy of not pulling any of this crap within a couple of months of an election. Bill Barr is absolutely dead-set on violating that policy. Why? Because that's how Trump won last time! Two weeks before the 2016 election, James Comey violated that policy, in the face of warnings from the rest of the DoJ, in order to announce a "re-opened investigation" into the Hillary Clinton email nonsense. Why? Someone found a laptop sitting around at the home of Anthony Weiner and Houma Abedin. All of the emails had already been checked, there was nothing there, Comey knew that, the re-opened investigation was bullshit, and as soon as Comey made the announcement, the polls moved "bigly." It was the only October surprise on record that actually moved the polls, and it did so by more than enough to swing the electoral college.
This... this is what Barr is planning. Well, not this precisely, but something like this. Barr is going to use the DoJ in a similar way to try to get late movement in the polls. Absolute, 100% guarantee.
And this matters for polling, for two reasons. First, the basic observation here about social science is that we can calculate "margins of error," which are based on sampling error. Get a sample of 700 or so, pretend it's random, and then because you could have gotten a weird sample, we estimate a 95% confidence interval such that we are 95% confident that an actual parameter falls within that range. But that's just sampling error. What's harder? Dealing with exogenous events. Big stuff that can come along and alter the landscape in some more fundamental way.
Coronavirus did that. We aren't likely to see anything that big, but stuff happens. In 2016, Comey happened.
And more complicatedly, he happened so close to the election that we didn't have very much post-Comey polling. Those "wrong" polls from 2016? Mostly pre-Comey. We just didn't have the time to do as much post-Comey polling as we needed to do. And there was volatility. Clinton's numbers dropped, then started to rebound, at the national level. State-level polling? We just couldn't do all of the state-level polling we needed to do. Point being, Comey didn't just screw up the entire 2016 election (which... he did). He screwed up our ability to poll the election.
To be sure, I just told you Barr is going to try the same thing. Of course, that may be tempered by our advanced knowledge that Barr will try it. Barr is about as subtle as an air raid klaxon.
So what does this mean? This, honestly, is harder. Once we know about a bias, we can account for it in our interpretation and analysis. An exogenous event, too late for polling to study? That's hard. Although technically, election interference from Barr will be an endogenous event rather than an exogenous event, for anyone being persnickety about econo-jargon.
Regardless, my general point remains that it is not as simple as saying that the polls were "wrong" in 2016, so ignore them. They were "biased," in 2016, but we can account for bias, knowing the bias. We must also be aware of the eventuality of an "October surprise," and be concerned about what happens if such an event occurs too late for us to poll for it.
Of course, none of this even begins to get at the difficulty of polling for an election that will be conducted largely via absentee balloting, nor the even more fundamental problem that we have a sitting president who has effectively announced his unwillingness to step down, preemptively declaring the entire process fraudulent.
So there's that. Yay?
Comments
Post a Comment