D.C.'s teacher compensation reform is working

Save IMPACT from superficial racial gap analysis

(Bill O'Leary/The Washington Post via Getty Images)

A bit over 10 years ago and following a few years of intense political fighting, Washington D.C. implemented a major reform to how its public schools operate.

There are a lot of details to the new system, but it basically combined “reform” ideas (weaker job protections for teachers, compensation based more on evaluations of teacher performance, standardized tests used as an evaluation metric) with an overwhelmingly Democratic jurisdiction’s willingness to spend money on stuff. As a result, teachers are now subject to the IMPACT evaluation system, but one of the main points of the system is to pay teachers bonuses. Overall, pay went up so that D.C. now has higher starting salaries for teachers than any of the 50 states, and in overall cash compensation, we’re in fourth place.

The chancellor who spearheaded those reforms, Michelle Rhee, was a very controversial figure not only locally but also nationally, where she sought and received a lot of attention. When former mayor Adrian Fenty lost his re-election bid in 2010, she was replaced by Kaya Henderson, a much more conciliatory figure who pursued similar policies and essentially consolidated the Rhee reforms.

And the overall trajectory of DCPS during this period was good.

Teacher retention rates improved, suggesting that whatever the controversies, DCPS became in practice a more attractive place to work. DCPS enrollment numbers also went up, which is a decent indication that in a city with a giant charter school sector, parents’ sense was that the schools were improving. And D.C.’s NAEP scores continued to rise, even as they fell in most of the country. None of that is proof that the reforms “worked,” but I do think it’s relevant context for thinking about them. When you’re looking at a school system with rising teacher retention, rising enrollment, and rising test scores, it’s not obvious that you’d want to rock the boat.

But when the District completed its review of the program earlier in August, it generated some troubling headlines. The Washington Post wrote that “D.C. Teacher Evaluation System Has Academic Benefits, But Is Racially Biased, New Study Finds.”

Did the study actually find that? I don’t think that it did, and the ability of these findings to generate that conclusion is an instance I think of how pure gap-ism can lead to bad analysis and potentially bad policy.

What the study found

What the study found is not that the evaluation system is biased, but that Black teachers had lower scores on average. Now, bias could be the reason for that, and certainly it’s useful to conduct the equity analysis and see if there’s evidence of bias.

As I said earlier, students’ standardized test scores are a big part of the evaluation metric. But in a concession to both teacher sentiment and general suspicion about high-stakes testing, the system also evolved to include direct human evaluation of the teachers. One benefit of that approach is that it can be more holistic. One downside of “holistic” assessments is that they can be a venue for bias to slip into the system. Some of the teachers I know speak Black English in casual conversation, and it’s easy for me to imagine a biased evaluator automatically marking that down as bad even though there is good reason to believe this is helpful in teaching kids how and why to use Standard American English in formal settings.

So are the assessments biased? Some facts from the report:

• 54% of DCPS teachers are Black.

• 70% of DCPS assessors are Black.

• Black teachers get lower scores on both the test-based and non-test assessments.

• White assessors give Black teachers higher scores than Black assessors give them.

If this is a question of “bias,” it’s a strange form of bias, in other words, in which the Black assessors are more biased than the white ones.

That doesn’t mean it’s not happening; there’s weird stuff in the world. But it’s a somewhat odd conclusion to jump to when a rival explanation seems clearly available in the report — pre-reform DCPS had very few white teachers, and the newer crop of teachers who’ve entered the system since the adoption of the reforms is both better and more racially diverse.

Teachers vary by age as well as race

Right here in the equity report, we see that there is a huge gap in the racial composition of the DCPS workforce based on age, with 62% of white teachers under 35 versus 40% of Black teachers.

Now just by the numbers, you have to assume very few of those under-35 teachers were employed by DCPS back in 2009 when the reforms were implemented.

And remember the goal of the reforms was to improve the quality of the teacher workforce through three mechanisms:

1. Make more people want to apply to be DCPS teachers by raising pay.

2. Make effective teachers more eager to keep teaching in D.C. and less likely to burn out.

3. Push a small minority of ineffective teachers out of the classroom.

If DCPS succeeded at achieving goal number one, you’d see two things happen. One is that the applicant pool would become more diverse (which in the D.C. context means more non-Black applicants) and the teachers hired would be above average relative to the existing stock of teachers.

That appears to be exactly what happened, and it’s not evidence of bias — it’s evidence of the system working as intended. They made DCPS a more attractive place to work, so the younger cohorts of teachers are more diverse and more skilled than the older ones.

But what about the impact on veteran teachers? Well, among teachers rated “effective” or “highly effective” (i.e., the kind of teachers the system is seeking to retain), the retention rates are very high across the board. They are especially high for Black teachers.

What’s more, the distribution of bonus money is perfectly equitable. The system’s effective Black teachers are being paid for their work and retained at high rates. The system is working.

It should also be said that 60% of DCPS students are Black, which is higher than the Black share of teachers and is in turn higher than the Black share of D.C. residents. And of course in D.C.m like in every city, there are more students than teachers. So when thinking about equity it’s important to keep in mind that making the schools better has disproportionate benefits for Black people.

The reforms appear to work

What I’ve been arguing thus far is that contrary to the charge of “bias,” the system appears to have a good deal of internal validity.

Retention is up, retention is up across racial groups, retention is particularly high among the teachers the evaluation framework says we should be trying to retain, effective teachers across racial groups are making more money, the applicant pool is getting more diverse, and the newer hires are higher-rated than the average legacy teacher. The subjective assessments are consistent with the test-based assessments, and the subjective assessments do not feature evidence of assessors punishing non-white ones.

But just because an evaluation system retains the teachers the system says should be retained doesn’t mean the system is a good idea. Perhaps it is misidentifying which teachers are the best ones and ought to be paid more.

So what do we know? Well, we know that 1% of teachers were ranked “ineffective,” and 96% of them were fired. An additional 3% were ranked “minimally effective,” and of them, 36% were fired. So we are talking about a tiny minority of the workforce being pushed out of the system, consistent with most people’s experience that teachers are mostly working hard and doing a good job. But what happens when those low-rated teachers leave? Well, the most important evidence on this comes from Melinda Arnot, Thomas Dee, Veronica Katz, and James Wyckoff, who studied DCPS turnover in a 2017 paper. They found that when a teacher who was rated low by IMPACT leaves, student achievement improves by 0.14 standard deviations in reading and 0.21 standard deviations in math.

By contrast, when a non-sanctioned teacher leaves (because she’s moving to another city, burned out on the job, or for whatever other reason), student achievement falls, albeit a small amount that fails the test of statistical significance.

This again appears to be a system that is working. The ineffective teachers are a tiny minority of the population, they are leaving at a higher rate, and when they leave the system it is good for kids. When the effective teachers leave, that’s bad for kids, but they are leaving at a lower rate.

Now I will happily admit that none of this evidence is absolutely ironclad. And none of it contradicts the mountains of evidence that out-of-school factors dominate in-school factors in terms of student performance. But even though out-of-school factors are most important, as long as you have schools that employ teachers, it seems like you should try to use a better teacher compensation system versus a worse one.

IMPACT deserves support

The new system has increased teacher pay, and it’s also improved teacher retention, suggesting that in practice most teachers like it.

The Washington Teachers Union, however, has been furiously opposed to it for years and continues to be weirdly preoccupied with job protection for a tiny minority of weak performers. WTU President Jaqueline Pogue Lyons reacted to the study with a written statement saying “it’s stunning that DCPS clings to a fatally flawed, unfair evaluation system when there are so many effective models that we could adapt that actually help teachers and students and aren’t punitive or cause anxiety.”

That’s unfortunate. But what’s really unfortunate is that, according to Perry Stein’s reporting for the Washington Post, Henderson’s DCPS chancellor successor Lewis Ferebee seems a little wobbly on this. She has two different quotes from him that point in somewhat different directions:

• “This system has worked for us, but we know it has imperfections. There are elements of systemic racism embedded in all systems and organizations. Having greater clarity of where those lines of disparities are allows us to be more responsive than we have in the past.”

• “We are not going to abandon a tool that has been helpful because there are some challenges with implementation. You deal with those challenges and continue on with the spirit of improvement and get better.”

I’m not going to say on the basis of this data that IMPACT is perfect or that nobody could come up with an idea for useful tweaks. But I think it’s a mistake to concede that there is evidence here of “systemic racism.” The specific hypothesis that Black assessors are giving biased assessments of Black teachers, meaning that the overrepresentation of Black people in the assessor population is disadvantaging Black teachers, is counterintuitive but worth trying to examine in more detail with some number-crunching.

The housing gap

The real systemic racism revealed in the report is the unsurprising-but-important fact that highly rated teachers are less likely to be working in the city’s high-poverty schools.

DCPS can and probably should try to remediate that in part by increasing the extra compensation available to experienced, high-rated teachers who go to work in high-poverty schools. But the underlying glaring issue for the city’s public schools is the sharp stratification of house prices along the lines of perceived school quality. In D.C., the rich people tend to be white and the poor people tend to be Black. But there is nothing stopping a Black millionaire from buying a house that’s zoned for the much-desired Alice Deal Middle School. And many white families can’t afford to live in neighborhoods where the average price of a three-bedroom home is well above a million dollars.

These house price dynamics are outside the control of the Chancellor, but because D.C. (like most American cities) auctions access to schools via the real estate market, they are inextricably linked.

The biggest thing the city could do to further combat “systemic” inequities in the school system would be to rezone everywhere for more housing and use the resulting tax revenue to build more schools while continuing to raise the entry-level base. That would both broaden access to the city’s best schools and also further improve recruiting by increasing the real, housing-adjusted value of teacher salaries.