It's Election Day!

And a plea to avoid pundits making bad statistical inferences

Well, today’s the day! There is not a lot to say at this point, and I don’t want to waste your time by pretending anyone wants to read about anything else today.

From a content planning standpoint, the difficult thing about this week is not only do we not know who’s going to win the election, we don’t even really know when we’ll know who won. Whatever happens, though, we’ll be rolling content out over the next few days. Brian and I will be live in the Substack app this evening, once enough is known to have something to talk about. That video will be on the site (and in the Politix feed) when it’s processed and available on Wednesday. I’ll do some chats (that’s also in the app).

We’ll see what happens. Stay posted.

What I want to leave you with today, though, is a brief plea: Please, for the love of God, do not draw strong inferences about American society based on exit poll cross tabs, and please do not trust anyone who does this.

I get the temptation! One very intriguing development during this election cycle has been signs of a racial realignment, with white voters becoming more Democratic and non-white voters becoming less Democratic. We also have evidence of a growing gender gap. People are going to want to know what’s up with this stuff as quickly as possible, and exit polls seem like a good way to do this.

Except they’re not. Over and above the challenge inherent in any statistical sampling exercise, the basic problem exit pollsters have is that they have no way of knowing what the electorate they are trying to sample actually looks like, but they do know who won the election. They end up weighting their sample to match the election results, which is good because otherwise you’d have polling error about the topline outcome, which would look absurd. But this weighting process can introduce major errors in the crosstabs.

For example, the 2020 exit poll sample seems to have included too many college- educated white people. That was a Biden-leaning demographic group, so in a conventional poll, it would have simply exaggerated Biden’s share of the total vote. But the exit poll knows the “right answer” for Biden’s aggregate vote share, so to compensate for overcounting white college graduates in the electorate, it has to understate Biden’s level of support within this group. That is then further offset by overstating Biden’s level of support within all other groups. So we got a lot of hot takes in the immediate aftermath of the election about Biden’s underperformance with white college graduates, which was fake, while people missed real trends, like Trump doing better with non-white voters.

To get the kind of data that people want exit polls to deliver, you actually need to wait quite a bit for more information to become available from the Census and the voter files about who actually voted. Eventually, Catalist produced its “What Happened in 2020” document, and Pew published its “Behind Biden’s 2020 Victory” report. But those take months to assemble, and unfortunately, conventional wisdom can congeal in the interim.

So just say no to exit poll demographic analysis!

What you can do pretty rapidly is look at geography. Not every African-American lives in a Black neighborhood, and not every person who lives in a Black neighborhood is African-American. But if Harris does worse than Biden did in the predominantly Black neighborhoods of Philadelphia, Milwaukee, Detroit, and Atlanta, that’s probably because she did worse with Black voters overall. To be clear, if you pitch this inference in a statistics class, your professor will yell “ecological fallacy!” Back in 2001, David Brooks wrote a book based on the observation that richer places were voting for Democrats now and seemed to draw the inference that this meant richer people were voting for Democrats. That was not true in 2000 or 2004 or 2008 or 2012 — the rich states leaned blue but the rich people leaned red. Since that time, Republicans have started doing better with poor people and Democrats have started doing better with rich people, so Brooks looks like a prophet. But the statistical inference was still bad!

That said, we have enough neighborhood level racial segregation in the United States that there are plenty of examples of places where it’s hard to explain a geographic shift as revealing anything other than a shift internal to an ethnic group.

Unfortunately, geography doesn’t tell us anything about gender. As best we can tell, women were 53 percent of the electorate in 2022 and as high as 55 back in 2008, but in 2020, 2018, 2016, and most other years, they’re 54 percent. In this case, if you just assume women are 54 percent of the electorate and weight your exit poll accordingly, that’s probably going to be correct. But if there, in fact, is an interesting gender skew to turnout this year (which there certainly might be), this method would, by definition, miss it and give you all kinds of bad numbers as a result. The temptation to make that assumption and charge away with the takes is going to be hard to avoid, but I really do think you should try. There’s a statistical regularity around women being 54 percent of voters, but that’s not a law of nature. The sad truth is it’ll just take time for us to know.