I am curious if the situation arises where research is modified to fit what the government wants to hear?
I am not thinking of situations where whole research documents are complete works of fiction, but rather where researchers make post-hoc edits to a methodology in order to ensure the resulting numbers fit the desired policy. This is rather akin to a scientist deciding that a given inconvenient piece of data 'is an outlier', or where they run a battery of statistical tests on data to find the one that gives the most flattering results.
Part of my job is in transport economics, and let me tell you everything I have described (and more) is absolutely routine. The freedom we have to monkey around with numbers to 'make it work' is extensive - future year projections for traffic growth, model simulations, assumptions galore. And our clients don't care, because they are the people who just want to bring good news to their bosses. Nobody actually sees anything wrong with this arrangement, it's just how things work.
So I'm wondering - does this ever happen? Who wants to be the person that tells the government that their new plan is not going to result in new jobs, or won't work?
Kolko's examples seem harmless, but one background concern I have is whether academics are harming academia by trying too hard to influence policy debates. There's a lot of folks out there who are doing stuff like "Historian here, here's why voting for Trump is exactly the same as supporting Hitler in 1932" on Twitter that makes academia look completely partisan and makes the public mistrust it. This is related to stuff like the public health guys' open letter saying science showed the lives saved by BLM protests would outweigh COVID deaths, which helped polarize COVID and did great harm to public health.
I don't think it is so great for academia to have lots of people there involved in the project of "how can I help Democratic Party politicians". It will impair needed credibility with the public. It's probably best if politicians used more think tank stuff and take what they can from academia without academics thinking in these terms.
The unglamorous work of collecting and counting data is generally more valuable than fancy theorizing, especially when said theorizing is unmoored by actual data.
I finished my PhD last year and then moved into government work, and this piece captured a lot of what I think is silly about academic research. At the end of the day the vast majority of it isn’t useful for policy work, no matter how many times they say it has important policy implications in the intro. But I agree that you can’t publish in a top journal or get tenure track doing most of the work you’re describing, unless you’ve already “made it.”
I’d be interested to see more ideas on what to do about policies that don’t end up working so well (or that are actually bad in practice). It seems as if it’s hard to unwind even these.
Just a shout out to two groups the author neglected who do practical econ policy research so well on food and ag: the USDA's Economic Research Service and at the universities, the Extension service.
On a related matter, there are economic data that the Government should produce:
1. Treasury should issue TIPS at less than 5 years tenor to see market expectation of inflation in the short term. Five- and Ten-year indicators are highly useful (I believe Larry Summers muscled Treasury into issuing them) but shorter terms are needed, too!
2. BLS needs to do real wage indexes. It does instead detailed unit value indexes of wage payments. The detail minimizes the problem but unit value indices are subject to composition effects. We do not do unit values for good and other services, we should not do them for wages.
3. Treasury (again) should issue a future GDP valued security, a "Trillionth." I would be very useful to know market expectations of GDP. We get some indication of the from stock market movements, but that is far inferior to an actual Trillionth. Issued at the same tenors as TIPS they would lend them selves to deducing expectations of real GDP movements.
Wait till he sees how useless most academic legal publishing is to practicing lawyers, judges, legislators, or really anyone else except other legal academics.
If the economists you worked with were there on a temporary basis, what kind of person provided the underlying support framework? All the long-term knowledge and relationships that would communicate policy goals into action?
What research economists could also use in my opinion is research into the accuracy of their data and how it affects the output of their data. This is particularly true for the big Wall Street economists who make predictions about inflation, employment, growth that often fail miserably.
Great guest. Interesting observation: "Cleaning up contaminated sites for redevelopment creates jobs at a much lower cost per job than job training, which in turn is much more cost-effective than giving businesses tax breaks or grants to create jobs."
I tend to hear this one about retraining diminished by cynicism from the plural of anecdotes: "There's no jobs for the retrained.","The trainees aren't interested in learning, they only want their old job back, they're just there to qualify for the check."
In any case, to the extent this data you say has checked out, I hoped it's not just accounted for and acted upon in federal programs, but widely circulated to states, counties, municipalities, and tribal entities, because the last, less efficient alternative of tax breaks and incentives shaves away fiscal health, and pits localities against each other in a race to the bottom and is an open invitation to self-dealing.
I appreciate the insights on what types of research are useful to government agencies for making policy, but there seems to be an implicit assumption, intended or not, of "it should be someone else's responsibility to make sure this research gets funded, completed, and published in a timely fashion." I get it's not easy to change how an institution like the Department of Commerce operates, but... no. If you need additional info to do your job well, then it is also your job to make sure that there is some sort of RFP with a timely process for getting the project scope, funded, and done to produce that info. That may very well mean funding academics, it may not, but either way it's not their responsibility to change how they operate based on your needs unless you make it worth their while. I know, I know, "Fix how government buys things it needs" is a quick way to get bogged down in an endless quagmire, but that doesn't make it less true.
I am curious if the situation arises where research is modified to fit what the government wants to hear?
I am not thinking of situations where whole research documents are complete works of fiction, but rather where researchers make post-hoc edits to a methodology in order to ensure the resulting numbers fit the desired policy. This is rather akin to a scientist deciding that a given inconvenient piece of data 'is an outlier', or where they run a battery of statistical tests on data to find the one that gives the most flattering results.
Part of my job is in transport economics, and let me tell you everything I have described (and more) is absolutely routine. The freedom we have to monkey around with numbers to 'make it work' is extensive - future year projections for traffic growth, model simulations, assumptions galore. And our clients don't care, because they are the people who just want to bring good news to their bosses. Nobody actually sees anything wrong with this arrangement, it's just how things work.
So I'm wondering - does this ever happen? Who wants to be the person that tells the government that their new plan is not going to result in new jobs, or won't work?
Kolko's examples seem harmless, but one background concern I have is whether academics are harming academia by trying too hard to influence policy debates. There's a lot of folks out there who are doing stuff like "Historian here, here's why voting for Trump is exactly the same as supporting Hitler in 1932" on Twitter that makes academia look completely partisan and makes the public mistrust it. This is related to stuff like the public health guys' open letter saying science showed the lives saved by BLM protests would outweigh COVID deaths, which helped polarize COVID and did great harm to public health.
I don't think it is so great for academia to have lots of people there involved in the project of "how can I help Democratic Party politicians". It will impair needed credibility with the public. It's probably best if politicians used more think tank stuff and take what they can from academia without academics thinking in these terms.
The unglamorous work of collecting and counting data is generally more valuable than fancy theorizing, especially when said theorizing is unmoored by actual data.
I finished my PhD last year and then moved into government work, and this piece captured a lot of what I think is silly about academic research. At the end of the day the vast majority of it isn’t useful for policy work, no matter how many times they say it has important policy implications in the intro. But I agree that you can’t publish in a top journal or get tenure track doing most of the work you’re describing, unless you’ve already “made it.”
Very interesting post, thanks! Pairs well with yesterday’s Hypertext: https://hypertextjournal.substack.com/p/the-world-is-hard-to-change
I’d be interested to see more ideas on what to do about policies that don’t end up working so well (or that are actually bad in practice). It seems as if it’s hard to unwind even these.
Why is the kind of obvious solution that the government should just have more staff researchers tasked with these sorts of things?
Like having non-partisan researchers seems about as relevant as say bill scores from cbo but yet we need to jawbone academics to get useful research?
Just a shout out to two groups the author neglected who do practical econ policy research so well on food and ag: the USDA's Economic Research Service and at the universities, the Extension service.
On a related matter, there are economic data that the Government should produce:
1. Treasury should issue TIPS at less than 5 years tenor to see market expectation of inflation in the short term. Five- and Ten-year indicators are highly useful (I believe Larry Summers muscled Treasury into issuing them) but shorter terms are needed, too!
2. BLS needs to do real wage indexes. It does instead detailed unit value indexes of wage payments. The detail minimizes the problem but unit value indices are subject to composition effects. We do not do unit values for good and other services, we should not do them for wages.
3. Treasury (again) should issue a future GDP valued security, a "Trillionth." I would be very useful to know market expectations of GDP. We get some indication of the from stock market movements, but that is far inferior to an actual Trillionth. Issued at the same tenors as TIPS they would lend them selves to deducing expectations of real GDP movements.
Here is my expanded plea:
https://thomaslhutcheson.substack.com/p/improvements-in-macroeconomic-data
“Biden-Harris administration,” really? Is that a thing now? I don’t ever remember reading “Trump-Pence administration, “ or “Obama-Biden”? Kamala has done such a good job protecting our border so I guess she deserves co-billing. https://www.cnn.com/2021/10/15/politics/kamala-harris-border-migration/index.html
Wait till he sees how useless most academic legal publishing is to practicing lawyers, judges, legislators, or really anyone else except other legal academics.
Anything that helps the government get better data is a great idea.
If the economists you worked with were there on a temporary basis, what kind of person provided the underlying support framework? All the long-term knowledge and relationships that would communicate policy goals into action?
I have nothing to add, but I appreciated this post because it was very "real world" - exactly what Slow Boring should be all about.
What research economists could also use in my opinion is research into the accuracy of their data and how it affects the output of their data. This is particularly true for the big Wall Street economists who make predictions about inflation, employment, growth that often fail miserably.
Great guest. Interesting observation: "Cleaning up contaminated sites for redevelopment creates jobs at a much lower cost per job than job training, which in turn is much more cost-effective than giving businesses tax breaks or grants to create jobs."
I tend to hear this one about retraining diminished by cynicism from the plural of anecdotes: "There's no jobs for the retrained.","The trainees aren't interested in learning, they only want their old job back, they're just there to qualify for the check."
In any case, to the extent this data you say has checked out, I hoped it's not just accounted for and acted upon in federal programs, but widely circulated to states, counties, municipalities, and tribal entities, because the last, less efficient alternative of tax breaks and incentives shaves away fiscal health, and pits localities against each other in a race to the bottom and is an open invitation to self-dealing.
I appreciate the insights on what types of research are useful to government agencies for making policy, but there seems to be an implicit assumption, intended or not, of "it should be someone else's responsibility to make sure this research gets funded, completed, and published in a timely fashion." I get it's not easy to change how an institution like the Department of Commerce operates, but... no. If you need additional info to do your job well, then it is also your job to make sure that there is some sort of RFP with a timely process for getting the project scope, funded, and done to produce that info. That may very well mean funding academics, it may not, but either way it's not their responsibility to change how they operate based on your needs unless you make it worth their while. I know, I know, "Fix how government buys things it needs" is a quick way to get bogged down in an endless quagmire, but that doesn't make it less true.