I am curious if the situation arises where research is modified to fit what the government wants to hear?
I am not thinking of situations where whole research documents are complete works of fiction, but rather where researchers make post-hoc edits to a methodology in order to ensure the resulting numbers fit the desired policy. This is rather akin to a scientist deciding that a given inconvenient piece of data 'is an outlier', or where they run a battery of statistical tests on data to find the one that gives the most flattering results.
Part of my job is in transport economics, and let me tell you everything I have described (and more) is absolutely routine. The freedom we have to monkey around with numbers to 'make it work' is extensive - future year projections for traffic growth, model simulations, assumptions galore. And our clients don't care, because they are the people who just want to bring good news to their bosses. Nobody actually sees anything wrong with this arrangement, it's just how things work.
So I'm wondering - does this ever happen? Who wants to be the person that tells the government that their new plan is not going to result in new jobs, or won't work?
Feel like that's where summaries of multiple analysis related to specific topics comes into play (sus out feasibility of some of the assumptions and sensitivity).
I once asked a senior National Institutes of Health official (the US health research funding agency), why they didn't fund more policy relevant work. Things that would help government officials leading Medicare and Medicaid to make better decisions. He was outraged, and said "we don't fund drug A versus drug B research." In the context what I think he meant was that the research isn't very interesting AND subtextually that the NIH did not want to get involved in anything controversial with high stakes. At that point the near death experience of the Agency for Health Care Policy Research at the hands of outraged back surgeons was very much in people's minds. That agency has been renamed Agency of Health Care Research and Quality. (Stay away from that policy stuff, its a career killer.)
I think that's where you require an after action review where you ascertain whether the policy produced the desired results. Ideally done by a different party
Kolko's examples seem harmless, but one background concern I have is whether academics are harming academia by trying too hard to influence policy debates. There's a lot of folks out there who are doing stuff like "Historian here, here's why voting for Trump is exactly the same as supporting Hitler in 1932" on Twitter that makes academia look completely partisan and makes the public mistrust it. This is related to stuff like the public health guys' open letter saying science showed the lives saved by BLM protests would outweigh COVID deaths, which helped polarize COVID and did great harm to public health.
I don't think it is so great for academia to have lots of people there involved in the project of "how can I help Democratic Party politicians". It will impair needed credibility with the public. It's probably best if politicians used more think tank stuff and take what they can from academia without academics thinking in these terms.
As I said, the examples seem harmless but I am saying if academics adopt the mindset of "how can I help Democratic administrations?" that could be bad.
Yeah my takeaway was - if you want ANY administration to make better decisions, here is how you can give them better info.
Maybe it'll be about something you personally hate (tariffs?) but maybe the research will help show which tariffs are most effective for the smallest pain and at least maybe that's what will get done.
I wonder what Karen Dunn Kelley, the only Senate-confirmed Under Secretary for Economic Affairs in Commerce under Trump would say about this post. She looks like a standard business-oriented Republican and it's possible that Trump-type brain infestation didn't penetrate to her level. I suspect she was not as interested in policy-related academic research like Kolko is, but did she shut the door entirely?
I think this is a little law brained. In other fields, policy research can be neutral, generally won’t be politically charged, and doesn’t look like advocacy. The law is different because I don’t perceive much legal scholarship as being anything other than advocacy pieces, which explains many other differences (lack of peer review, student editors, etc.).
Huh? Whether researchers who read this take Kolko's advice has no bearing on whether pop-scientists do partisan hot takes on Twitter, or in the scientific literature for that matter. (if you think his advice *causes* partisan hot takes, I'm bewildered since you made no attempt to argue that case.)
The unglamorous work of collecting and counting data is generally more valuable than fancy theorizing, especially when said theorizing is unmoored by actual data.
I finished my PhD last year and then moved into government work, and this piece captured a lot of what I think is silly about academic research. At the end of the day the vast majority of it isn’t useful for policy work, no matter how many times they say it has important policy implications in the intro. But I agree that you can’t publish in a top journal or get tenure track doing most of the work you’re describing, unless you’ve already “made it.”
I didn’t say the purpose of Econ academia is to be useful for policy work. I said much of it claims to be useful for policy work and it isn’t. And I find that aspect of it to be silly. If the field was a little self aware, then I would agree more with what you’re saying.
I’d be interested to see more ideas on what to do about policies that don’t end up working so well (or that are actually bad in practice). It seems as if it’s hard to unwind even these.
Seems to me that one way to do this would be to give academics more academic incentives to do this kind of work. For instance, if government users cited academic work in ways that show up in academic citation counts, that could help.
The Becker Model was developed in the context of biomedical research partly for this reason, to give a more holistic portrait of impact than just citations: https://becker.wustl.edu/impact-assessment/model
The best places in government for producing policy research are the Fed banks, as mentioned in the article. They're also an anomaly in having independent revenue and a substantial degree of control over their own funding and staffing decisions. Maybe that's a clue?
Just a shout out to two groups the author neglected who do practical econ policy research so well on food and ag: the USDA's Economic Research Service and at the universities, the Extension service.
On a related matter, there are economic data that the Government should produce:
1. Treasury should issue TIPS at less than 5 years tenor to see market expectation of inflation in the short term. Five- and Ten-year indicators are highly useful (I believe Larry Summers muscled Treasury into issuing them) but shorter terms are needed, too!
2. BLS needs to do real wage indexes. It does instead detailed unit value indexes of wage payments. The detail minimizes the problem but unit value indices are subject to composition effects. We do not do unit values for good and other services, we should not do them for wages.
3. Treasury (again) should issue a future GDP valued security, a "Trillionth." I would be very useful to know market expectations of GDP. We get some indication of the from stock market movements, but that is far inferior to an actual Trillionth. Issued at the same tenors as TIPS they would lend them selves to deducing expectations of real GDP movements.
There are already 1 year TIPS available in the secondary market. What benefit would an auction have over the current market mechanism?
The very first TIPS auction on January 29 in 1997 was for 10 years, so I doubt Larry Summers had to strong arm anyone into anything. It was the first choice.
Five year TIPS were introduced a few months later. They were the second tenor. Again, seems doubtful any strong arming was required since it was the second choice.
30 year notes weren't introduced until the following year.
Given the pace at which Treasury does things, I imagine TIPS were in progress for a few years before January 1997, when Summers would have been an even more marginal figure.
When you work in the administration, it’s respectful and considered good practice to refer to both the President and the VP as the name of the administration. Think you might be reading into this one a bit much.
Google shows that the phrase “Biden Administration” is 67 times more frequent than “Biden-Harris Administration.” So it’s a little weird, but not unprecedented.
You must not have been paying attention to the 2020 primary. Biden called it the “Obama-Biden administration all the time; “Biden-Harris” is a continuation of that.
Wait till he sees how useless most academic legal publishing is to practicing lawyers, judges, legislators, or really anyone else except other legal academics.
Do Supreme Court justices, who say what the Constitution is, care at all what all the brilliant constitutional experts in law schools have to say, other than when it confirms their priors?
As a current student staffer on a law journal…legal scholarship is some of the most pointless I’ve ever seen. Legal academia is desperately overdue for some disruption.
As a counterpoint to what I said above, here is an example of academic legal publishing of a type that could be useful to lawmakers (judicial and legislative), because it provides an empirical description of the current law on the ground, in a controversial and contested area where there are competing claims about what and how stable the current law actually is. And where some of those in the debate have an incentive to overstate or misconstrue current law to gain support for legislation to change the law. Note it was not published in a law school law review.
I think there is a LOT of low-hanging fruit out there if legal scholars are willing to get more empirical. But they face serious headwinds such as: it’s hard to get data on courts; law tends to draw people from non-technical backgrounds who are uncomfortable with/distrustful of empiricism; student-run journals (and the profession more broadly) value prestigious credentials over content; some legal philosophies explicitly reject empiricism; etc.
If the economists you worked with were there on a temporary basis, what kind of person provided the underlying support framework? All the long-term knowledge and relationships that would communicate policy goals into action?
What research economists could also use in my opinion is research into the accuracy of their data and how it affects the output of their data. This is particularly true for the big Wall Street economists who make predictions about inflation, employment, growth that often fail miserably.
Great guest. Interesting observation: "Cleaning up contaminated sites for redevelopment creates jobs at a much lower cost per job than job training, which in turn is much more cost-effective than giving businesses tax breaks or grants to create jobs."
I tend to hear this one about retraining diminished by cynicism from the plural of anecdotes: "There's no jobs for the retrained.","The trainees aren't interested in learning, they only want their old job back, they're just there to qualify for the check."
In any case, to the extent this data you say has checked out, I hoped it's not just accounted for and acted upon in federal programs, but widely circulated to states, counties, municipalities, and tribal entities, because the last, less efficient alternative of tax breaks and incentives shaves away fiscal health, and pits localities against each other in a race to the bottom and is an open invitation to self-dealing.
I don't think Kolko is making this mistake, but many commenters seem to, so I will correct it. The need for the kind work described here is not in opposition to the kind of work that often gets published and is less directly useful for government. In fact, literature review and modeling are both reliant on prior literature. The problem is not that novel work that requires a PhD to appreciate isn't valuable, it's that we also need to value the work described here.
I am curious if the situation arises where research is modified to fit what the government wants to hear?
I am not thinking of situations where whole research documents are complete works of fiction, but rather where researchers make post-hoc edits to a methodology in order to ensure the resulting numbers fit the desired policy. This is rather akin to a scientist deciding that a given inconvenient piece of data 'is an outlier', or where they run a battery of statistical tests on data to find the one that gives the most flattering results.
Part of my job is in transport economics, and let me tell you everything I have described (and more) is absolutely routine. The freedom we have to monkey around with numbers to 'make it work' is extensive - future year projections for traffic growth, model simulations, assumptions galore. And our clients don't care, because they are the people who just want to bring good news to their bosses. Nobody actually sees anything wrong with this arrangement, it's just how things work.
So I'm wondering - does this ever happen? Who wants to be the person that tells the government that their new plan is not going to result in new jobs, or won't work?
Thankfully, we have the CBO. Which, to my knowledge, appears objective and nonpartisan.
https://www.youtube.com/watch?v=3M7SzS_5PlQ
Feel like that's where summaries of multiple analysis related to specific topics comes into play (sus out feasibility of some of the assumptions and sensitivity).
I once asked a senior National Institutes of Health official (the US health research funding agency), why they didn't fund more policy relevant work. Things that would help government officials leading Medicare and Medicaid to make better decisions. He was outraged, and said "we don't fund drug A versus drug B research." In the context what I think he meant was that the research isn't very interesting AND subtextually that the NIH did not want to get involved in anything controversial with high stakes. At that point the near death experience of the Agency for Health Care Policy Research at the hands of outraged back surgeons was very much in people's minds. That agency has been renamed Agency of Health Care Research and Quality. (Stay away from that policy stuff, its a career killer.)
I think that's where you require an after action review where you ascertain whether the policy produced the desired results. Ideally done by a different party
Kolko's examples seem harmless, but one background concern I have is whether academics are harming academia by trying too hard to influence policy debates. There's a lot of folks out there who are doing stuff like "Historian here, here's why voting for Trump is exactly the same as supporting Hitler in 1932" on Twitter that makes academia look completely partisan and makes the public mistrust it. This is related to stuff like the public health guys' open letter saying science showed the lives saved by BLM protests would outweigh COVID deaths, which helped polarize COVID and did great harm to public health.
I don't think it is so great for academia to have lots of people there involved in the project of "how can I help Democratic Party politicians". It will impair needed credibility with the public. It's probably best if politicians used more think tank stuff and take what they can from academia without academics thinking in these terms.
This is a misread of the article IMO. It seemed to be encouraging more actionable policy research, not more political advocacy by researchers.
As I said, the examples seem harmless but I am saying if academics adopt the mindset of "how can I help Democratic administrations?" that could be bad.
Doing relevant research is, or at least should be, useful for all administrations. The fact Dems are in the WH is incidental to the argument IMO.
Yeah my takeaway was - if you want ANY administration to make better decisions, here is how you can give them better info.
Maybe it'll be about something you personally hate (tariffs?) but maybe the research will help show which tariffs are most effective for the smallest pain and at least maybe that's what will get done.
I wonder what Karen Dunn Kelley, the only Senate-confirmed Under Secretary for Economic Affairs in Commerce under Trump would say about this post. She looks like a standard business-oriented Republican and it's possible that Trump-type brain infestation didn't penetrate to her level. I suspect she was not as interested in policy-related academic research like Kolko is, but did she shut the door entirely?
I think this is a little law brained. In other fields, policy research can be neutral, generally won’t be politically charged, and doesn’t look like advocacy. The law is different because I don’t perceive much legal scholarship as being anything other than advocacy pieces, which explains many other differences (lack of peer review, student editors, etc.).
Huh? Whether researchers who read this take Kolko's advice has no bearing on whether pop-scientists do partisan hot takes on Twitter, or in the scientific literature for that matter. (if you think his advice *causes* partisan hot takes, I'm bewildered since you made no attempt to argue that case.)
The unglamorous work of collecting and counting data is generally more valuable than fancy theorizing, especially when said theorizing is unmoored by actual data.
It resonated with me when the author swooned over people who update their datasets monthly.
"That Works Very Well in Practice, But How Does It Work In Theory?" - Old economists joke.
I finished my PhD last year and then moved into government work, and this piece captured a lot of what I think is silly about academic research. At the end of the day the vast majority of it isn’t useful for policy work, no matter how many times they say it has important policy implications in the intro. But I agree that you can’t publish in a top journal or get tenure track doing most of the work you’re describing, unless you’ve already “made it.”
I don't think the main purpose of academic work is to be useful for policy work. It's supposed to be about advancing the field and making discoveries.
Mostly it fails at that too, but that's because in every human endeavor, 80% is crap, 19% is okay, 0.9% is damn good and 0.1% is outstanding.
I didn’t say the purpose of Econ academia is to be useful for policy work. I said much of it claims to be useful for policy work and it isn’t. And I find that aspect of it to be silly. If the field was a little self aware, then I would agree more with what you’re saying.
Academia in general really needs to get fucked. One of the best arguments for abolishing tenure is that then there would be no need to pursue it!
If that’s one of the best arguments then I shudder to think of how shoddy the rest are.
I’m surprised you’re not already aware of them. They’ve been in the news for years.
I'm surprised you're not already aware of all the counterarguments. They've been in the news for years.
I’m not the one wondering what they are, my guy.
Very interesting post, thanks! Pairs well with yesterday’s Hypertext: https://hypertextjournal.substack.com/p/the-world-is-hard-to-change
I’d be interested to see more ideas on what to do about policies that don’t end up working so well (or that are actually bad in practice). It seems as if it’s hard to unwind even these.
Why is the kind of obvious solution that the government should just have more staff researchers tasked with these sorts of things?
Like having non-partisan researchers seems about as relevant as say bill scores from cbo but yet we need to jawbone academics to get useful research?
There's a huge amount of money and effort going into academic research - we may as well try to make it useful.
Seems to me that one way to do this would be to give academics more academic incentives to do this kind of work. For instance, if government users cited academic work in ways that show up in academic citation counts, that could help.
The Becker Model was developed in the context of biomedical research partly for this reason, to give a more holistic portrait of impact than just citations: https://becker.wustl.edu/impact-assessment/model
All hail the H-Index.
Academics are in many respects government researchers. Might as well get something useful for the taxpayers’ money.
The best places in government for producing policy research are the Fed banks, as mentioned in the article. They're also an anomaly in having independent revenue and a substantial degree of control over their own funding and staffing decisions. Maybe that's a clue?
But they also are tasked with real decision making and the quality of their decisions will eventually be evaluated.
Congress should have more researchers and policy wonks on staff.
Just a shout out to two groups the author neglected who do practical econ policy research so well on food and ag: the USDA's Economic Research Service and at the universities, the Extension service.
I don't think he neglected them; I think he was talking particularly from the perspective of Commerce.
Thanks. Yes, I chose my words poorly. Thank you for the correction.
On a related matter, there are economic data that the Government should produce:
1. Treasury should issue TIPS at less than 5 years tenor to see market expectation of inflation in the short term. Five- and Ten-year indicators are highly useful (I believe Larry Summers muscled Treasury into issuing them) but shorter terms are needed, too!
2. BLS needs to do real wage indexes. It does instead detailed unit value indexes of wage payments. The detail minimizes the problem but unit value indices are subject to composition effects. We do not do unit values for good and other services, we should not do them for wages.
3. Treasury (again) should issue a future GDP valued security, a "Trillionth." I would be very useful to know market expectations of GDP. We get some indication of the from stock market movements, but that is far inferior to an actual Trillionth. Issued at the same tenors as TIPS they would lend them selves to deducing expectations of real GDP movements.
Here is my expanded plea:
https://thomaslhutcheson.substack.com/p/improvements-in-macroeconomic-data
There are already 1 year TIPS available in the secondary market. What benefit would an auction have over the current market mechanism?
The very first TIPS auction on January 29 in 1997 was for 10 years, so I doubt Larry Summers had to strong arm anyone into anything. It was the first choice.
Five year TIPS were introduced a few months later. They were the second tenor. Again, seems doubtful any strong arming was required since it was the second choice.
30 year notes weren't introduced until the following year.
Given the pace at which Treasury does things, I imagine TIPS were in progress for a few years before January 1997, when Summers would have been an even more marginal figure.
“Biden-Harris administration,” really? Is that a thing now? I don’t ever remember reading “Trump-Pence administration, “ or “Obama-Biden”? Kamala has done such a good job protecting our border so I guess she deserves co-billing. https://www.cnn.com/2021/10/15/politics/kamala-harris-border-migration/index.html
When you work in the administration, it’s respectful and considered good practice to refer to both the President and the VP as the name of the administration. Think you might be reading into this one a bit much.
Google shows that the phrase “Biden Administration” is 67 times more frequent than “Biden-Harris Administration.” So it’s a little weird, but not unprecedented.
You must not have been paying attention to the 2020 primary. Biden called it the “Obama-Biden administration all the time; “Biden-Harris” is a continuation of that.
There were, shall we say, strong electoral considerations attending to Biden's gluing himself to Obama during the 2020 campaign.
Yep!
Wait till he sees how useless most academic legal publishing is to practicing lawyers, judges, legislators, or really anyone else except other legal academics.
Do Supreme Court justices, who say what the Constitution is, care at all what all the brilliant constitutional experts in law schools have to say, other than when it confirms their priors?
Akhil Amar is very frustrated that they don’t listen to him! And at least in Trump v. Anderson he is right!
As a current student staffer on a law journal…legal scholarship is some of the most pointless I’ve ever seen. Legal academia is desperately overdue for some disruption.
As a counterpoint to what I said above, here is an example of academic legal publishing of a type that could be useful to lawmakers (judicial and legislative), because it provides an empirical description of the current law on the ground, in a controversial and contested area where there are competing claims about what and how stable the current law actually is. And where some of those in the debate have an incentive to overstate or misconstrue current law to gain support for legislation to change the law. Note it was not published in a law school law review.
https://patentlyo.com/patent/2023/11/predictability-framework-perspective.html
This is excellent!
I think there is a LOT of low-hanging fruit out there if legal scholars are willing to get more empirical. But they face serious headwinds such as: it’s hard to get data on courts; law tends to draw people from non-technical backgrounds who are uncomfortable with/distrustful of empiricism; student-run journals (and the profession more broadly) value prestigious credentials over content; some legal philosophies explicitly reject empiricism; etc.
Anything that helps the government get better data is a great idea.
If the economists you worked with were there on a temporary basis, what kind of person provided the underlying support framework? All the long-term knowledge and relationships that would communicate policy goals into action?
I have nothing to add, but I appreciated this post because it was very "real world" - exactly what Slow Boring should be all about.
What research economists could also use in my opinion is research into the accuracy of their data and how it affects the output of their data. This is particularly true for the big Wall Street economists who make predictions about inflation, employment, growth that often fail miserably.
“…big Wall Street economists who make predictions about inflation, employment, growth that often fail miserably”
If that were the case, those big economists would be unemployed.
They don’t get fired if they are wrong along with everyone else. If you’re wrong in way that’s different from your peers - that gets you fired.
You’ll also get fired for telling the powers that be things they don’t want to hear.
I doubt that, but whatever.
Don’t you remember all the economists predicting the recession that never came? Are they still employed? Of course.
Such as…?
Oh Ken, you sweet innocent child.
Nothing succeeds like failure in our America.
Great guest. Interesting observation: "Cleaning up contaminated sites for redevelopment creates jobs at a much lower cost per job than job training, which in turn is much more cost-effective than giving businesses tax breaks or grants to create jobs."
I tend to hear this one about retraining diminished by cynicism from the plural of anecdotes: "There's no jobs for the retrained.","The trainees aren't interested in learning, they only want their old job back, they're just there to qualify for the check."
In any case, to the extent this data you say has checked out, I hoped it's not just accounted for and acted upon in federal programs, but widely circulated to states, counties, municipalities, and tribal entities, because the last, less efficient alternative of tax breaks and incentives shaves away fiscal health, and pits localities against each other in a race to the bottom and is an open invitation to self-dealing.
I don't think Kolko is making this mistake, but many commenters seem to, so I will correct it. The need for the kind work described here is not in opposition to the kind of work that often gets published and is less directly useful for government. In fact, literature review and modeling are both reliant on prior literature. The problem is not that novel work that requires a PhD to appreciate isn't valuable, it's that we also need to value the work described here.