1) I am in favor of accountability in education, and the most important form of accountability is merely providing an account. Students, parents and the communities that schools serve should be provided an account of their schools' performances. (in many things. Yes, at teaching core academic lessons and advanced academic lesson, but also children's mental health and emotional development. I care less about sports program performance, though that has long been the easiest to find.)
2) I am even in favor of test-based accountability. Eventually, students should show what they can do without the scaffolding and support of their teachers. That's the rubber meeting the road.
3) The problem, as I've been telling Matt for over a decade, is that the tests do not measure what they claim to measure. Matt listed the definition of "basic" reading skills for the NAEP. But he provided no evidence that the tests actually measure that. The advanced word for this is "instrumentalization." That is, turning the construct definition into an instrument of measurement. And I'm telling you, as an expert on this topic, the tests suck. They are bad. They do not measure what they claim to measure.
4) On the NAEP in particular—and it's a test with some incredibly good design elements—there's a particular problem. The construct definitions for the NAEP do NOT match the learning standards which states require schools to teach towards. They were written before states generally had high level standards (i.e., expectations and learning goals) that were so similar. The logic of the effort to raise state standards and make them more similar was to make it easier to construct tests that could measure school quality across states. But NAEP never adopted the new construct definitions—which are no longer new. NAEP prioritized the consistency of what they were measuring over fidelity to state standards and curriculum.
Over time, as schools get better and better at teaching the next gen standards, we would expect NAEP scores to drop—or at least remain stagnant.
5) So, how precise can our inferences from NAEP scores be? Well, I don't trust small drops as being meaningful at all. And I *know* that what tests measure is not actually what they claim to report on.
6) What I see is that the tests are too focused on more simplistic and contextless cognition, which is precisely what next gen standards aim to more beyond. So, honestly, it is disturbing that students are having trouble with these (in my view) lower level skills. But that doesn't mean that they are not at the same time making progress on more advanced and generalizable higher level skills. We just don't know.
7) You want a shorter version of this? A man drops his house keys at night. Instead of searching inside his dark home where he lost them, he looks under a nearby streetlamp. When asked why, he replies, “Because the light is better here.” Just because we get quantitive data from these tests doesn't mean that the answers we want are anywhere in there.
Sigh. I came from a country that worshiped the US and its education system, and it's only after becoming simultenously a parent and a teacher of teenagers that I start to see American students are...NOT SO SMART...almost intentionally made so by their teachers in a lot of cases. And the teachers, they were taught by education school to do so. As an immigrant who worked in the US for 20 years in tech and finance before becoming a teacher, I found it so eye-opening that I wrote a book describing my experiences. If anyone is interested, my sustack has info on it.
"But when you hold schools accountable for results at the bottom, they have no choice but to pay attention to instruction methods that work, which has positive results for basically all students."
I wish this were true. I agree with Matt that the kids who thrive in the current system are those who would likely thrive in any system, but based on my research (I'm an education writer and the author of two books on cognitive science and literacy), I have to disagree that a regime of test-based accountability worked for all kids. In reading, test scores have essentially been stagnant since 1998. NCLB made a little difference in its early years, but not much.
There's a lot I could say about what's wrong with our education system (like, at least two books' worth), but briefly, here's why test-based accountability hasn't worked, at least in the area of reading: The tests purport to measure abstract comprehension skills, like "making inferences," putting pressure on teachers to focus on those skills rather than on any particular content. But comprehension skills aren't abstract, transferable skills like riding a bike or decoding words. Cognitive science tells us that background knowledge is far more important to comprehension than "skill."
Kids who come from more highly educated families are better able to pick up the academic knowledge and vocabulary that enables reading comprehension, which is a major reason for their better performance on reading tests. The other kids depend on school for that kind of information. And especially in a system that values lots of practice in comprehension "skills" over building substantive academic knowledge, they're the least likely to get it there.
Test-based accountability only makes sense if kids are being tested on stuff they've actually been taught. But standardized reading tests essentially test them on stuff they HAVEN'T been taught. And more test-based accountability, at least the way reading tests are currently formulated, will only make the situation worse for kids at the lower end of the socioeconomic spectrum.
How much of the recent decline in education is due to AI? Both generative AI and the algorithmic feed? I mean a lot of the effect is masked by the pandemic but I do think the timing of ChatGPT is suggestive here.
At the time, there was concern with “teaching to the test” - an example of Goodhart’s Law, where when measure becomes a target, it ceases to be a good measure. Was there something to this critique?
The conclusion that American students are getting dumber is unwarranted until someone shows that scores can be compared between years. A similar claim was made based on the PISA data, but that's publicly available so we were able to see that the scores are not, in fact, comparable, and that incomparability drives the apparent negative trends.
I’m also concerned about this. But if we’re going to bring back accountability for test scores, we have to set realistic goals and make sure that reading and math don’t devour all other subjects.
NCLB set an unrealistic goal of 100% proficiency, and then threatened schools with punishment if they failed to eventually meet it. Lots of teachers concluded that the real goal of the law was to delegitimize public schools to pave the way for privatization. Many schools stopped teaching subjects other than reading and math and did nothing but teach to the test, because they were so afraid of punishment.
We need to learn from those mistakes and do better this time.
Just switched back to using mostly paper in my middle school classes. Having assignments on computers was easier and worked for about 10 years, but then the AI- and Google-use became second nature to them and too few would bother to think or write without it. Computers are almost like cellphones with distractions since students can always find something other than classwork.
I wish it weren't impossible to move kids along based on proficiency rather than grade level. My happiest time in school was when we did a reading program where you could move at your own pace. The craziest time was when I read whole books during geometry class and could still answer every question the teacher posed. Can't say that was time wasted though.
It’s hardly the only reason, but some schools don’t assign as many novels to their students due to block scheduling. Some configurations of the schedule mean that students only have a major course for half of a year. The longer block allows for the same number of hours in the classroom, but because most reading is done outside of school, you can’t assign the same number of books to read from September through January as you could from September to June.
Thank you, Matt Y, for an informative albeit depressing piece.
How does all of it interact with the Flynn Effect? Did we just run into the natural limitations of the Flynn effect, or is the Flynn effect getting swamped by some deleterious factor like smartphone use frying kids' brains?
Matt notes that schools are teaching "passages" (excerpts) of literature, instead of forcing students to read entire books. He suggest that this is cost-based (presumably, the schools copy/share online a few paragraphs rather than buying books).
This seems like an ideal opportunity for a public-private partnership. Given the state of the book business, it would be wise for publishers or authors to donate mass copies of The Great Gatsby (or another of a hundred books typically taught in schools). This is especially true if they can do it digitally at essentially zero cost.
Teaching the next generation to *actually read books* would seem imperative to publishers, even if they lose the handful of sales to schools that still actually assign full books. (FWIW, my sophomore came home with instructions that I needed to provide a copy of an assigned book by the following day, which sent me on a mad scramble across area libraries).
A couple other factors that I think may explain some of the NCLB backlash besides what Matt listed:
1) NCLB was poisoned by its association with George W. Bush. A lot of liberals probably assumed that the stated equity goal was fake, and the policy was going to be used as a pretext for diverting money from struggling public schools to private Christian schools.
2) People believed NCLB was ineffective due to Goodhart’s law (“when a measure becomes a target it ceases to be a good measure”). In other words, people felt that test scores were an artificial reflection of student growth since teachers were incentivized to “teach to the test.” And they felt that students’ actual critical thinking skills / motivation / etc. were suffering because they were being taught rote memorization and test taking rather than being inspired to learn and grow.
Right or wrong, I think these concerns may have played a larger role in the backlash than parents not wanting to hear that their kids did poorly on tests.
It also set a goal of 100% proficiency, which was obviously not realistic. Lots of schools were so desperate to avoid punishment for below-par test scores that they axed all subjects other than reading and math. Of course teachers and parents hated that.
It’s always a bit shocking how much of everything is just keeping people from doing things other than their job (he says, instead of tackling an annoying task pile).
1) I am in favor of accountability in education, and the most important form of accountability is merely providing an account. Students, parents and the communities that schools serve should be provided an account of their schools' performances. (in many things. Yes, at teaching core academic lessons and advanced academic lesson, but also children's mental health and emotional development. I care less about sports program performance, though that has long been the easiest to find.)
2) I am even in favor of test-based accountability. Eventually, students should show what they can do without the scaffolding and support of their teachers. That's the rubber meeting the road.
3) The problem, as I've been telling Matt for over a decade, is that the tests do not measure what they claim to measure. Matt listed the definition of "basic" reading skills for the NAEP. But he provided no evidence that the tests actually measure that. The advanced word for this is "instrumentalization." That is, turning the construct definition into an instrument of measurement. And I'm telling you, as an expert on this topic, the tests suck. They are bad. They do not measure what they claim to measure.
4) On the NAEP in particular—and it's a test with some incredibly good design elements—there's a particular problem. The construct definitions for the NAEP do NOT match the learning standards which states require schools to teach towards. They were written before states generally had high level standards (i.e., expectations and learning goals) that were so similar. The logic of the effort to raise state standards and make them more similar was to make it easier to construct tests that could measure school quality across states. But NAEP never adopted the new construct definitions—which are no longer new. NAEP prioritized the consistency of what they were measuring over fidelity to state standards and curriculum.
Over time, as schools get better and better at teaching the next gen standards, we would expect NAEP scores to drop—or at least remain stagnant.
5) So, how precise can our inferences from NAEP scores be? Well, I don't trust small drops as being meaningful at all. And I *know* that what tests measure is not actually what they claim to report on.
6) What I see is that the tests are too focused on more simplistic and contextless cognition, which is precisely what next gen standards aim to more beyond. So, honestly, it is disturbing that students are having trouble with these (in my view) lower level skills. But that doesn't mean that they are not at the same time making progress on more advanced and generalizable higher level skills. We just don't know.
7) You want a shorter version of this? A man drops his house keys at night. Instead of searching inside his dark home where he lost them, he looks under a nearby streetlamp. When asked why, he replies, “Because the light is better here.” Just because we get quantitive data from these tests doesn't mean that the answers we want are anywhere in there.
Sigh. I came from a country that worshiped the US and its education system, and it's only after becoming simultenously a parent and a teacher of teenagers that I start to see American students are...NOT SO SMART...almost intentionally made so by their teachers in a lot of cases. And the teachers, they were taught by education school to do so. As an immigrant who worked in the US for 20 years in tech and finance before becoming a teacher, I found it so eye-opening that I wrote a book describing my experiences. If anyone is interested, my sustack has info on it.
"But when you hold schools accountable for results at the bottom, they have no choice but to pay attention to instruction methods that work, which has positive results for basically all students."
I wish this were true. I agree with Matt that the kids who thrive in the current system are those who would likely thrive in any system, but based on my research (I'm an education writer and the author of two books on cognitive science and literacy), I have to disagree that a regime of test-based accountability worked for all kids. In reading, test scores have essentially been stagnant since 1998. NCLB made a little difference in its early years, but not much.
There's a lot I could say about what's wrong with our education system (like, at least two books' worth), but briefly, here's why test-based accountability hasn't worked, at least in the area of reading: The tests purport to measure abstract comprehension skills, like "making inferences," putting pressure on teachers to focus on those skills rather than on any particular content. But comprehension skills aren't abstract, transferable skills like riding a bike or decoding words. Cognitive science tells us that background knowledge is far more important to comprehension than "skill."
Kids who come from more highly educated families are better able to pick up the academic knowledge and vocabulary that enables reading comprehension, which is a major reason for their better performance on reading tests. The other kids depend on school for that kind of information. And especially in a system that values lots of practice in comprehension "skills" over building substantive academic knowledge, they're the least likely to get it there.
Test-based accountability only makes sense if kids are being tested on stuff they've actually been taught. But standardized reading tests essentially test them on stuff they HAVEN'T been taught. And more test-based accountability, at least the way reading tests are currently formulated, will only make the situation worse for kids at the lower end of the socioeconomic spectrum.
How much of the recent decline in education is due to AI? Both generative AI and the algorithmic feed? I mean a lot of the effect is masked by the pandemic but I do think the timing of ChatGPT is suggestive here.
At the time, there was concern with “teaching to the test” - an example of Goodhart’s Law, where when measure becomes a target, it ceases to be a good measure. Was there something to this critique?
The conclusion that American students are getting dumber is unwarranted until someone shows that scores can be compared between years. A similar claim was made based on the PISA data, but that's publicly available so we were able to see that the scores are not, in fact, comparable, and that incomparability drives the apparent negative trends.
https://x.com/cremieuxrecueil/status/1953992171201380516
I’m also concerned about this. But if we’re going to bring back accountability for test scores, we have to set realistic goals and make sure that reading and math don’t devour all other subjects.
NCLB set an unrealistic goal of 100% proficiency, and then threatened schools with punishment if they failed to eventually meet it. Lots of teachers concluded that the real goal of the law was to delegitimize public schools to pave the way for privatization. Many schools stopped teaching subjects other than reading and math and did nothing but teach to the test, because they were so afraid of punishment.
We need to learn from those mistakes and do better this time.
Just switched back to using mostly paper in my middle school classes. Having assignments on computers was easier and worked for about 10 years, but then the AI- and Google-use became second nature to them and too few would bother to think or write without it. Computers are almost like cellphones with distractions since students can always find something other than classwork.
I wish it weren't impossible to move kids along based on proficiency rather than grade level. My happiest time in school was when we did a reading program where you could move at your own pace. The craziest time was when I read whole books during geometry class and could still answer every question the teacher posed. Can't say that was time wasted though.
Do we have polling on parental involvement in education? Is it possible their lack of concern is what drives all of these other trends?
Nothing is going to work if parents just don't care that much about good grades.
It’s hardly the only reason, but some schools don’t assign as many novels to their students due to block scheduling. Some configurations of the schedule mean that students only have a major course for half of a year. The longer block allows for the same number of hours in the classroom, but because most reading is done outside of school, you can’t assign the same number of books to read from September through January as you could from September to June.
Agree fully. 220 days.
Thank you, Matt Y, for an informative albeit depressing piece.
How does all of it interact with the Flynn Effect? Did we just run into the natural limitations of the Flynn effect, or is the Flynn effect getting swamped by some deleterious factor like smartphone use frying kids' brains?
Matt notes that schools are teaching "passages" (excerpts) of literature, instead of forcing students to read entire books. He suggest that this is cost-based (presumably, the schools copy/share online a few paragraphs rather than buying books).
This seems like an ideal opportunity for a public-private partnership. Given the state of the book business, it would be wise for publishers or authors to donate mass copies of The Great Gatsby (or another of a hundred books typically taught in schools). This is especially true if they can do it digitally at essentially zero cost.
Teaching the next generation to *actually read books* would seem imperative to publishers, even if they lose the handful of sales to schools that still actually assign full books. (FWIW, my sophomore came home with instructions that I needed to provide a copy of an assigned book by the following day, which sent me on a mad scramble across area libraries).
It's the publishers doing this. They don't want to pay royalties for full books.
A couple other factors that I think may explain some of the NCLB backlash besides what Matt listed:
1) NCLB was poisoned by its association with George W. Bush. A lot of liberals probably assumed that the stated equity goal was fake, and the policy was going to be used as a pretext for diverting money from struggling public schools to private Christian schools.
2) People believed NCLB was ineffective due to Goodhart’s law (“when a measure becomes a target it ceases to be a good measure”). In other words, people felt that test scores were an artificial reflection of student growth since teachers were incentivized to “teach to the test.” And they felt that students’ actual critical thinking skills / motivation / etc. were suffering because they were being taught rote memorization and test taking rather than being inspired to learn and grow.
Right or wrong, I think these concerns may have played a larger role in the backlash than parents not wanting to hear that their kids did poorly on tests.
It also set a goal of 100% proficiency, which was obviously not realistic. Lots of schools were so desperate to avoid punishment for below-par test scores that they axed all subjects other than reading and math. Of course teachers and parents hated that.
It’s always a bit shocking how much of everything is just keeping people from doing things other than their job (he says, instead of tackling an annoying task pile).