The known unknowns of our hot button debates

Nobody really knows how many people got murdered last year or what's taught in schools

(Craig F. Walker/The Boston Globe via Getty Images)

I’ve been thinking lately about the way the basic structures of journalism encourage us to write around the fact that there are big obvious questions that we don’t know the answers to; that which the late Donald Rumsfeld might have called the “known unknowns” of the policy debate. And I think it actually might be a good idea to write more explicitly about this stuff — especially here on the blog where we’re not constrained by normal journalism conventions.

After all, I totally understand why nobody wants to run a front-page story headlined “Nobody Knows How Many People Were Murdered Last Year.”

But we have had a lot of stories about various individual murders, and stories about murder trends, and stories about the politics of crime, and stories about crime control policy, and meta-stories about the impact of media coverage on perceptions of crime.

Given the relatively large amount of attention paid to the issue, it seems like some attention should be paid to the fact that we don’t really know how many people were murdered last year and that our understanding of non-murder crimes is even worse.

Sometimes the government knows things

Sometime in late September or early October of 2021, the FBI should release “Crime in the United States 2020.”

That will give us an “official” count of how many murders, car thefts, burglaries, robberies, rapes, and other crimes occurred last year. And then if we’re pouring over those numbers in October 2021 and want to know whether things got better or worse over the summer of 2021, we’ll have to wait all the way until the fall of 2022.

By contrast, I can give you very precise information about the chicken situation as of May 2021 — there were 851 million broiler-type chickens hatched (a 5% increase from the previous May) and 57.3 million egg-laying chickens hatched (a 9% increase from the previous May). American chickens laid a total of 9.38 billion eggs in May (up 3%), of which 8.07 billion are table eggs (i.e., you’re going to eat them), 1.23 billion were broiler-type hatching eggs, and the balance were egg-layers.

We don’t get real-time data on the chicken situation, but it’s close. The May poultry report I got those numbers from came out on June 22.

So it takes something like three to four weeks to get updated on chickens but over nine months to get updated on homicides.

Lots of economic data come out with this fairly rapid cadence. On Friday we got the Employment Situation Report for the previous month and it was a big news story, as the monthly ESR almost always is. One price we pay in order to get that jobs data so frequently and so quickly is that it’s based on statistical sampling rather than an actual count, and the numbers tend to get revised.

But with murders, we’re in a much worse state. Instead, what happened is that back in March we got the FBI’s quarterly report, which basically just sums up reports from a relatively small number of big cities. Then experts who are accustomed to working with these numbers know from experience that national crime trends are usually pretty uniform, so we can probably just project a 25% increase across the board.

Jeff Asher @Crimealytics

So murder was likely up 25-30% nationally in 2020. A 25% increase in murder from 2019 would mean there were over 20,000 murders in the U.S. for the first time since 1995.

Note, though, that this is not a scientifically devised sampling method. It’s just literally that some jurisdictions report early and others report late, so the FBI gives you the early numbers and analysts assume that trends will be uniform. I think that’s probably correct. But in this case, it leaves us high and dry, because the question of whether it’s actually true that the increase was uniform is central to contemporary debates.

Was the 2020 murder surge uniform?

We know that the pandemic was global, but the 2020 murder surge was not global.

So while the pandemic is obviously part of the context for the 2020 murder surge, it’s not a sufficient explanation for it. The other thing that happened in 2020 was a massive wave of racial justice protests that tended to take a specifically anti-police angle, so maybe rising murder is related to anti-police activism.

The most simple version of this debate begins and ends with the observation that relatively few cities actually did enact cuts to police budgets, and we know the surge in lethal violence extended well beyond that set.

The more interesting argument is whether a broader set of left-of-center politicians in some sense induced de-policing by adopting anti-police rhetoric. Or to frame it the other way: did police officers stage a massive de facto strike in every jurisdiction that made any gesture at accountability? However you want to characterize the de-policing hypothesis, you’d want to start with looking at some source of variation in murder trends. But of course, you can’t do that if you’re just assuming that the early-reporting cities are representative of the national trend.

And then we end up with the problem that there’s very little political variation among America’s big cities.

• Jacksonville has a Republican mayor and Florida has a Republican governor, but murders surged 22% last year.

• On the other hand, El Paso is one of the few cities that saw murders fall in 2020, and they had a Republican mayor at the time (he lost in 2020 and now the mayor is a Democrat), and Texas of course has a Republican governor.

You could spin this sparse data either way if you wanted to, but the obvious problem is just that the data is too sparse. By contrast, if you actually had complete homicide statistics, there are lots of interesting questions you could ask.

• Most cities are part of a county. Did murder go up more in those cities than in the counties that surround them?

• Did murder go up more in states with Democratic governors, or does it not matter?

• If you look at suburban counties, did murder go up more in counties with GOP-run county commissions? District attorneys?

• We know there are lower murder rates in rural areas, but did murders increase there specifically?

• Given the centrality of race to the arguments about policing, do we see increases specifically in counties with a larger Black population share?

• Does the increase in murders correlate with some index of Covid restrictions? Maybe restaurant closures took eyes off the street?

At any rate, don’t even ask me what the best questions to ask about this hypothetical data are. Real social scientists are going to be much more aware of the pitfalls and the theories and ways to explore them. It’s just obvious that you could shed light on a lot of debates if we actually had the information about how many people were murdered and where, but we do not.

Did crime go up or down last year?

My personal best guess is that the increase was in fact fairly uniform, and we are going to struggle to find a political or policy determinant of the problem. I look at the FAA’s reports of an increased level of aggressive and unruly behavior by airline passengers and think that we are looking at some kind of nationwide epidemic of bad behavior, of which the shootings are just the tip of the iceberg.

But this is actually at odds with another talking point from the criminal justice reform community, which is that crime is actually down in most reporting cities — it’s only murders that are increasing.

That’s very unusual, as normally the crime aggregates move in tandem. It’s not particularly surprising that we saw a decline in burglaries during a year when people were at home an unusual amount. And you could maybe explain a decline in muggings the same way. But why would there be fewer assaults at the same time as there are more murders? And how is it that all the tech bros in San Francisco are convinced there’s been a surge in shoplifting and petty crime when the data says the opposite?

Maybe we’ve got a mystery on our hands. Or maybe we have a decline in reporting.

Traditionally, homicide is the crime to look at if you want to do inter-city or inter-temporal comparisons because the reporting of dead bodies is relatively uniform while the reporting on other kinds of crimes gets very fuzzy. I do think it’s troubling that alongside murder, the other crime we’re seeing more of is stolen cars — that’s a crime that the victim almost always reports for insurance and registration issues. Maybe part of the Floyd fallout was a big decline in citizen disposition to have confidence in the police, which simultaneously caused a decline in reports of small crimes and an increase in the actual occurrence of serious ones.

Again, it might be interesting to test this against a deep and rich set of data covering diverse jurisdictions, but we don’t have that. After all, it’s not something important like up-to-date info on chicken hatchery volumes.

What’s being taught in school?

The other big raging debate that has glaring data quality problems is recent culture wars about the content of K-12 education.

What Wesley Yang says here is, I think, pretty clearly what most people believe.

Wesley Yang @wesyang

Easy: 1. Teach the history of slavery, civil war, Reconstruction, Jim Crow, and the Civil Rights Movement 2. Don't teach that white teachers "spirit murder" black and brown kids, that objectivity is white supremacy, that "other ways of knowing" are just as valid as reason

Now of course there are people with extreme views on both sides. But the reason this fight is so bitter and nasty even though most people don’t hold extreme views is that there’s a lot of disagreement as to what’s actually happening.

• Factually speaking, it is true that there is an education professor at the University of Georgia named Bettina Love who does takes about “How Schools Are ‘Spirit Murdering’ Black and Brown Students,” and she’s done trainings in San Diego and is presumably teaching to future teachers and administrators in her Georgia gig.

• Factually speaking, it is also true that there’s an eighth-grade history textbook in use in Louisiana that appears to teach the Civil War and Reconstruction exclusively through the eyes of white slaveowners.

There are 130,000 K-12 schools in the United States of America, so you could do five stories per week about the excesses of the left-most 1% of the schools, each of which had five example schools. And you could do the exact same with the excesses of the most conservative schools. And you might still be left with a situation in which actually 98% of schools are totally fine and sensible.

What’s more, there’s what’s on paper and then there’s what’s happening in classrooms.

In my experience, lots of people sit through weird DEI trainings, roll their eyes, and then proceed as normal. It’s just a checkbox compliance thing from the legal department. The problem is that the DEI trainers themselves aren’t in on the joke, so they’ve developed some elaborate pseudo-radical ideology. And by the same token, it’s very normal for history teachers to go off-book if there’s something in the textbook they don’t think is spun the right way.

Now don’t get me wrong — schools shouldn’t use bad textbooks and districts shouldn’t be assigning bizarre trainings. But it would still be nice to know the actual scale of these issues. Yet there’s no way to actually know what’s happening.

Information — the ultimate public good

I sometimes poke fun at America’s incredibly precise agricultural statistics because it seems so random. Why do I need to know that from January to May of 2021 we produced more veal but less lamb and mutton than we produced from January to May of 2020?

But that’s the glory of these stats. Who knows why I might need to know that? Anyone who does want to know it can look it up very easily. You don’t even need to know where to look it up; just google something like “lamb production statistics” and you’ll find the USDA website very quickly.

Raw factual information of that sort is a classic public good — non-rivalrous, non-excludable — and therefore something that it’s very beneficial to the public to produce. And it’s great that our government publishes so much of it. It’s unfortunate, though, that there are major lacunae, including on big hot button issues. I’m not sure there’s any feasible way to get the answers I want on education. But on murder, setting up a program where every homicide logged by every county coroner goes into a federal database in a timely manner seems very achievable.

Public goods are also a good place for the non-profit and philanthropical sectors to step in and fill gaps in what the government provides.

But unfortunately, as best I can tell, descriptive work is low-prestige across broad swathes of academia. Not everywhere, of course. One area I’ve been reading about more is paleoanthropology and paleogenetics, where the hot thing is to find new bones and say what they look like or to extract DNA from some bones and do a fresh analysis of the genomes. In more contemporary periods, though, just counting stuff wouldn’t necessarily be considered a great project. And from a strictly scholarly point of view, the fact that we won’t find out how many people were murdered this year until October isn’t the end of the world. Some good research about the 2020 murder surge will circulate in working paper form in 2022 and get published someday.

This is fine as far as it goes, but there’s actually a lot of public utility in having reasonably timely statistical information available to the public. Instead of flailing around in the dark on these issues or doubling down on arguments based on incomplete data, we ought to be trying to actually improve the quantity of information available to us.