More good news about D.C.'s teacher pay reforms

The new system is good, but it all comes back to housing in the end

(Marvin Joseph/The The Washington Post via Getty Images)

I want to follow up on two points readers made in response to last week’s post, D.C.’s teacher compensation reform is working, because it turns out there’s more to say about this.

The first is that the post didn’t address the real issue with the current IMPACT system: the massive stress it puts on the city’s teacher workforce. The second is that it didn’t talk enough about how gentrification and peer effects factor in.

D.C.'s teacher compensation reform is working

On the first, I should have been clearer that one of the strengths of the research on the reforms I cited is that it shows this isn’t a big problem in practice. There are complaints about the system, but retention is up, and when teachers leave that the city wanted to retain, they do not cite the system. The union is understandably very responsive to its noisiest members, but I think the media should avoid overemphasizing the views of the most discontented minority.

On the second, another reader helpfully shared that I missed the biggest and most rigorous assessment of the reforms — a Mathematica Policy Research brief from August 12 that is very quantitatively sophisticated and also looks at these things through a wider lens.

To make a long story short, Mathematica shows that the improvement in student achievement cannot be attributed to an influx of higher-income students. Such an influx did occur, but the authors find improvements to student learning — especially in math — over and above what you can attribute to either the compositional effect of a higher-SES student population or the peer effect of students having more high-SES kids in class with them.

This is important because there are significant feedback effects between the housing and education market. The impact of school finance changes on student demographics could act as a force multiplier for change by generating better, more integrated schools for all, or it could undermine them by squeezing poor families out of neighborhoods with improving schools.

D.C.’s reforms have improved retention

Any time you change something that affects a large number of people, some of them are going to be unhappy with it. I have certainly heard from DCPS teachers who don’t like the new system.

To review what I covered in the prior post, the basic tenets of the new system include:

• higher base pay

• bonus pay for the highest-rated teachers

• reduced job security for the lowest-rated teachers

The goal is to raise average pay and average teacher quality by making DCPS more attractive to new applicants and more competitive at retaining the best teachers, while also encouraging a small minority of teachers to move on.

This summer, DCPS released a multi-faceted review of the IMPACT system that included an ethnography-style review by a team of American University professors who did interviews with 46 teachers. They say their “sample represents a broad swath of DCPS teachers across a diverse range of identities and experiences,” but critically they don’t say it’s a random sample. A random sample of a diverse population should itself be diverse, but just because your sample is diverse doesn’t mean it’s random or representative.

What they find is that “overall perceptions of IMPACT were more negative than positive” due to “an unhealthy environment of distrust, fear, and competitiveness in schools that trickles down into the classroom.” The researchers, to their credit, note that spring 2020 was a stressful time and maybe not the ideal moment to survey people about their subjective feelings about things.

But more broadly, I question the relevance of these findings. I’ve never heard of a workforce where there isn’t some grousing about performance assessments. The question is whether, all things considered, the new package has improved the city’s ability to recruit and retain good teachers. A separate report focused specifically on this says the answer is yes. Most notably, during a period when the national labor market conditions were steadily improving, the share of teachers that DCPS retained went steadily up within both specific schools and the system as a whole.

That’s a key sign that, whatever 46 people told some folks from AU, DCPS has become a better, not worse place to work. The report also notes that “new teachers hired by DCPS outperform separated and non-retained teachers on their final IMPACT score,” and they repeat the finding from Adnot et al. that I cited last week showing that when low-scoring teachers leave, their students start doing better.

And since people who don’t like the system are raising spurious racial equity complaints, it’s worth noting that retention of Black and Latino teachers is better than retention of white teachers.

Only 7% of teachers who leave DCPS cite dissatisfaction with the evaluation system as the reason. This genuinely is much higher than the national average of 2%, but it’s still extremely low. Among teachers who are rated effective or highly effective, by far the largest reason for leaving is relocation to another city (number two is retirement). This is one of several examples of the interplay between education and housing that I discuss in more detail below.

But here’s the key point: you can’t improve schools with a “performance pay” system if it makes good teachers not want to work there. But DCPS has become a more attractive place to work, and that’s improved the schools!

Mathematica Policy Research’s findings

Dallas Dotter, Duncan Chaplin, and Maria Bartlett’s report for Mathematica is very quantitatively sophisticated and lets us speak very precisely about how student achievement has improved in D.C. over the past 10 years.

They use data that goes way back to the early 1990s to get a clear sense of changes in the underlying trend (NAEP scores were going up even before the reforms) and to construct a better counterfactual. Here are their key findings:

• The reforms led to better-than-expected gains in Grade 4 Math and Reading NAEP scores.

• By Grade 8, the reading improvement has phased out and is back at the baseline, but the Grade 8 math improvements are still present (apparently it is typical to find that in-school changes have a larger impact on math).

• There are bigger changes in student performance for kids who were exposed to more years of reform.

• The data is not yet good enough to get a statistically credible estimate of high school performance.

This is a good reminder that it’s unrealistic to expect fiddling with the school system to generate large student impacts. The gains described above are very large compared to gains linked to other school reforms, similar to the total restructuring of New Orleans Public Schools after Hurricane Katrina or to the most optimistic estimates of shrinking class size. But they’re just not that big. School performance is dominated by out-of-school factors. The reason to care about education reform and related topics isn’t that futzing with school management is going to revolutionize American society; it’s that as long as a city is running a school system, it should try to run it as well as possible — even if the scale of the potential impacts is limited.

Speaking of which, it’s reasonable to look at these DCPS changes and ask whether we should in fact attribute them to increased gentrification in D.C. The improvements occurred during a time in which, after several decades of population loss, the city began to grow again, with growth dominated by higher-income people. Most of the newcomers are childless young professionals, but there are some of us aging yuppies with kids now enrolled in DCPS. An influx of richer kids with better-educated parents is going to mechanically push up test scores.

Mathematica looks at this and they find that, yes, if you just eyeball the NAEP scores, you will overestimate the success of the education reform. They think about 40% of the raw score increase is accounted for by shifting student demographics — both a compositional shift toward more students with higher test scores and also the positive impact of a somewhat less segregated system.

But the results I reported in the bullet points above are what you get after you adjust for that. In other words, Grade 8 reading scores are in fact higher, but Mathematica thinks that goes away when you adjust for demographics and that actually, just math in Grades 4 and 8 and reading in Grade 4 improved. And when you do the demographic analysis, it turns out that the improvements attributable to reform are concentrated among Black students.

So, again, the scale of the school improvements here is not large enough to put them forward as a panacea to major social problems. But it is good to have better schools, and looking at these reforms through an equity lens strengthens rather than weakens the case for them.

Schools and gentrification

Because the Mathematica researchers are fussy, they are at pains to distinguish the causal effect of the teacher compensation reforms from changes attributable to peer effects. The idea is that more high-SES kids coming into a school can have spillover benefits to the other kids, and this is different from better teaching.

But from a policy viewpoint, I think we should see this as a flywheel.

High-SES parents are very sensitive to school quality and also probably overestimate the scale of genuine peer effects relative to what most researchers find. So if you implement a reform that genuinely improves student learning, the visible test scores will go up. That will induce more high-SES students to enroll. That will generate some further improvements in student learning (via peer effects and spillovers), as well as some apparent improvements via the compositional effect. And that in turn will induce even more high-SES students to enroll.

To an extent, this is good. We want schools (and neighborhoods) to be less stratified by race and socioeconomic status, and the line between “gentrification” with its negative connotations and “integration” with its positive connotations can be pretty fuzzy.

But as with any other localized improvement in conditions, the real issue is what happens to the housing market. Right now the neighborhood that I live in is quite expensive, but the local middle school is not very well regarded and hardly any rich professionals send their kids there. If the school got better, that would be great for the whole community, including the low-income families who rely on it. But the price of housing would also go up, which is great for incumbent homeowners but bad for renters — especially low-income renters.

Now, I don’t think cities should maintain bad schools as an affordable housing policy. But this is a kind of general problem in local government.

There’s a classic 2013 Washington Post story about neighborhood resistance to a tree-planting effort that was regarded by the locals as a harbinger of gentrification and displacement. And the tragedy of a constrained housing market is this isn’t a purely irrational view that just reflects distrust or a lack of engagement. Under current conditions in D.C., anything you do to make a neighborhood a better place to live — better mass transit, better schools, a new park, nice trees, less pollution, safer streets — risks setting off a cycle of displacement. I don’t think that’s a reason to give up on improving city services. But it underscores the extent to which housing policy is fundamental to doing anything on an equitable basis. Unless the supply of homes can increase in line with demand, you are stuck in zero-sum scrambles for opportunity.