This and a few other posts over the years are very technical and wonderful as references. They belong in some kind of Slow Boring technocratic archive. Everything You Wanted to Know But Were Too Bored to Ask
"Also, limitations on data sharing across the 13 principal statistical agencies (yes, 13!) reduce the quality of federal statistics, introduce inconsistencies across different statistical products, and create duplicative work."
IFF it was interested in government efficiency. It's interested, if anything but generating Twitter content, in reducing expenditures and state capacity.
A concerted 4 year initial effort to start making government more effective woud be a great idea for a Democratic administration that wants government of do things. If you basically want a government that reduces taxes of high income people and keep foreign good and people out, you don't need an efficient government. And if you are planning on ruling by decree, you positively do not want one.
Musk and Ramaswamy are desperate for attention and affirmation. It is why they have both gravitated to MAGA world. I don't believe they are there because of any deeply held beliefs, as witnessed by the recent commentary around the H1B Visa program.
I think the best way to react to their silly DOGE thing is to try to co-opt them by trying to direct their focus onto real areas to improve government efficiency.
Understandable from my perspective, and obviously, someone has to address those issues, even if I’d prefer it to be someone else. Regardless, we need to fix something here, and I’m confident they’ll handle it in a way at least partially aligned with my values and interests.
We need a bipartisan commission with both tax increases and spending cuts. If they were concerned with efficiency then they would basically audit all sorts of processes and ask why it take agencies so long to act on things they are mandated to do (they could look at how slow IRA and BIL funding rolled out.)
The real reason is risk aversion and how we have given people too many opportunities to sue pretextually. This is something Congress could reform and narrow.
I wholeheartedly agree, but there is no appetite for serious tax increases among Republicans or serious spending cuts among Democrats. Pretty sure we're going to need either inflation/bond markets to enforce fiscal discipline on the US.
The data sharing that he's referring to mostly have to do with restrictions when working with individual-level data that must be analyzed within a federal research data center. The number of statistical agencies isn't really a problem unless you think BLS should start compiling crime data.
I’m trying to withhold complete cynicism that there’s anything going to happen with DOGE other than headline-grabbing “waste”, rather than promoting actual best practices for effective government.
1) When they move in a direction I don't like, it's because my political enemies are perpetrating a fraud to cover their incompetence.
2) When they move in a direction I do like, it's because a fraud perpetrated by my political enemies has been exposed and the record is being set straight.
I have great trust in our government statistical agencies. They are incredibly professional.* I have no trust in the Trump administration when it comes to keeping its hands off them when/if the numbers start turning sour.
Jed notes that political interference is one of the risks to data integrity. I hope in his next monthly post he updates us on what level of risk we're starting to face from Trump appointees, and continues to report on that as the administration unfolds.
*FRED is such a treasure that perhaps it should be considered for the economics Nobel? It has done more for broad-based understanding of the economy than any number of recent prize winners.
Right, there are things that could be better on the margin with our statistical agencies (many of these things would require Congress to act fwiw), but honestly we should all be proud of the data our government makes publicly available. These agencies (and sub-agencies) a great example of competent government doing things that are tremendously valuable that most people don't think about that often.
Informative article. However, while the author worries about "Inappropriate political interference," how we currently measure race and ethnicity (perhaps among other measures) is inherently political. The current categories are somewhat incoherent, mixing skin color (white, black) with geographic origins (Middle East and North Africa). Political interest groups (mainly progressive) have an outsized impact on its measurement.
Race and ethnicity don't have coherent definitions, and vary within cultural context. Different parts of the world define what is/isn't in these categories and how they interact with one another differently. Creating an 'objective' definition isn't possible. And yet, given the realities of the world, it would be a total abdication of responsibility to simply fail to measure these constructs. Not being able to differentiate African Americans from Whites in your data set isn't acceptable because it's very hard to create a conceptually satisfying framework.
Additionally, because of the long time-series feature of government statistics and their use by third parties (like me!) to benchmark and use for analysis, the government is shackled by an immense weight of history. The announced update to the OMB standards - collapsing Hispanic from an ethnicity into a race, and including MENA as a race separate from White - is the first update since 1997.
I'm not sure who you think defines these questions, but US statistical agencies take this stuff seriously. They do a ton of research and validation testing. And I'm not sure which progressive interest groups you think are advocating for changing things once every 30 years, but I don't know them.
Headcount. Age. Sex. Number of people in the household and relatedness. Type of dwelling. Own or rent. Citizenship. Immigration status. Employment status.
Yes, US statistical agencies take this seriously, and they do considerable testing, but the fact is that advocacy groups have significant influence on the process. Most changes made in racial classification increase the count of members of advocacy groups (which inevitably reduce the count of the white population). Here's one book that discusses some of these issues, from a fairly sympathetic viewpoint: https://journals.sagepub.com/doi/full/10.1177/0002716218756629
An interesting read, and definitely merits more time than I can give it now. I would just note that advocates can make bad arguments for good outcomes, and considering the socially constructed and ever-shifting nature of these constructs, sometimes the experts are going to agree through other sources of knowing.
Maybe this is naive, and perhaps we're already doing it, but when an objective measurement isn't possible, one solution is to support multiple measurements.
One thing I don't understand is that we're just fine with outmoded surveys and revisions up to a year later. We have the data itself across agencies to track this stuff in near real time if we wanted to. The entire private sector world has been investing into CDPs that organize their customer data across online and offline touchpoints, in real time, for the use of ads ROI for over a decade now. Working with general survey data like this is a policy choice to be inaccurate in the age of immediately available data.
Yep. I’m a fan of making everyone participate in these things fully. Get a new job, report it to the state. Get a raise, report it to the state. For W2 employees, the Work Number aggregates this info anyway! I’ve only had one W2 job in my whole life that isn’t on there, and I only worked there 3 months. Even one of my 1099 jobs is on there!
I hope at the very least our compilers of official statistics have data as solid as the Work Number’s.
I read your post linked below, and I think it is worth thinking seriously about differences between how the private sector uses and collects data versus the public sector. There are technological limitations in the public sector, but there also legal limitations that prevent how federal agencies can collect or use data. Frankly, the private sector is much less constrained than the public sector, and a lot of the "modernizing" required would need Congress to update laws like the Privacy Act, as one example.
These limitations are policy choices made without an iota of provable security benefits. The constraints are self-imposed and would not require congress to update any laws at all in order to start building CDP-like capabilities out of the reams of non-PII digital data available. E.g. See the slow rollout of login.gov, that should simply be mandated to use across the board for every agency that touches constituents.
The sheer amount of granular interactivity data held within the GSAs DAP program (https://analytics.usa.gov/) most Fortune 500 companies would die for, and it's mostly gone totally unused for work outside basic annual program analysis of user counts and volume
I think we agree, but might be talking about slightly different things or focusing on different components For some things that rely surveys, for example, it's not clear to me what alternatives exist for collecting individual-level data or in real-time. For example, the Survey of Consumer Finances (https://www.federalreserve.gov/econres/scfindex.htm) collects information from households that couldn't be collected otherwise.
I think this is different from a question of how the government collects and uses data more broadly. The government does not provide a great customer experience. Basically, both these are true:
1. The government does not manage or use the data it collects effectively
2. The government faces limitations that make it different from the private sector in what it can and cannot do
Sure, we do need legislation for specific survey data collection regimes, I agree with that. My broader point is that the survey of consumer finances is probably less accurate than a model of consumer finances based on inputs that a CDP could control -- macro data, user applications to specific gov benefits programs, more detail into bank data they are required to submit, etc. There are different ways to measure these things and the over reliance on surveys opens up all of these quantitative issues that, measured via other means, would be far less of a concern.
Wouldn’t relying on job postings result in a significant overestimate of available jobs? It costs a company nothing to post a position they may not really intend to fill.
Aaron, you are a respected data analyst and a long time user of Census data. I'm very curious as to why you see the group quarters imputation in response to disruption of normal procedures by the pandemic as "cooking the books". (Context is that they improvised an unplanned procedure in 2020 because due to the pandemic, they had group quarters that showed as occupied in January/February 2020 in their Group Quarters Advanced Contact, but provided no data or inconsistent data during July-October follow ups). This affected 2% of all Group Quarters nationwide. The American Statistical Association independent assessment of Census data quality in August 2021 found GQ imputation as adding minimal risk at the state level, though local risk would be higher. The Bureau published a detailed report in 2023 of their methodology. https://www2.census.gov/library/publications/decennial/2020/2020-census-group-quarters-imputation.pdf You can disagree with the approach, but why would you call it "cooking the books:"?
Congressional apportionment is based on the number of persons in a state, including persons who cannot legally vote such as minors, legal permanent residents, felons who have had their voting rights stripped, etc. I’ve seen people argue that apportionment should be based only on the number of registered voters or only on the voting-age (and voting-eligible) population but under current law that’s not the way things have been set up.
i mean the constitution pretty clearly just says "free persons" (excluding Indians as Ken points out) so excluding children or non-citizens would require a constitutional amendment.
The reason it damages official statistics is that controversies around immigrants and the Census make it harder for Census to get immigrants to respond to surveys. Source: getting immigrants to respond to surveys was my job at one time.
It's here as a thing to be linked to from social media when folks who don't understand how and why there are revisions to official statistics get freaked out when they find out that there was a revision to an official statistic.
I think the fear is Trump will start cooking official statistics while claiming he is correcting past impropriety, as he did with election administration.
My understanding is there's at least four moving parts in every report: (1) payroll survey, (2) household survey, (3) household adjustments, (4) seasonal adjustments and Sacks misunderstood which was what.
The sad thing is that any econ grad student could tell the tech guys here why they're wrong, but the tech guys aren't willing to rise to the level of competence of a mediocre grad school.
This and a few other posts over the years are very technical and wonderful as references. They belong in some kind of Slow Boring technocratic archive. Everything You Wanted to Know But Were Too Bored to Ask
"Everything You Wanted to Know But Were Too Bored to Ask" This would be great on a T-shirt (or other SB merch)!
Noted!
"Also, limitations on data sharing across the 13 principal statistical agencies (yes, 13!) reduce the quality of federal statistics, introduce inconsistencies across different statistical products, and create duplicative work."
Attention DOGE: Look for stuff like this.
IFF it was interested in government efficiency. It's interested, if anything but generating Twitter content, in reducing expenditures and state capacity.
A concerted 4 year initial effort to start making government more effective woud be a great idea for a Democratic administration that wants government of do things. If you basically want a government that reduces taxes of high income people and keep foreign good and people out, you don't need an efficient government. And if you are planning on ruling by decree, you positively do not want one.
Musk and Ramaswamy are desperate for attention and affirmation. It is why they have both gravitated to MAGA world. I don't believe they are there because of any deeply held beliefs, as witnessed by the recent commentary around the H1B Visa program.
I think the best way to react to their silly DOGE thing is to try to co-opt them by trying to direct their focus onto real areas to improve government efficiency.
I'd like to see a scintilla of evidence that they have any interest in making government work better.
They don't, but maybe it will grab their attention, like populating Mars
DOGE only cares about 1337 issues like FFA airspace restrictions and NHTSA’s inquiries into the safety of autonomous driving systems.
Understandable from my perspective, and obviously, someone has to address those issues, even if I’d prefer it to be someone else. Regardless, we need to fix something here, and I’m confident they’ll handle it in a way at least partially aligned with my values and interests.
My point is the Eye of Sauron isn’t focused on rent seeking rather than government efficiency
Yeah, we’re in agreement on our predictions. We only differ in our prioritization and possibly our judgment on the possibilities.
We need a bipartisan commission with both tax increases and spending cuts. If they were concerned with efficiency then they would basically audit all sorts of processes and ask why it take agencies so long to act on things they are mandated to do (they could look at how slow IRA and BIL funding rolled out.)
The real reason is risk aversion and how we have given people too many opportunities to sue pretextually. This is something Congress could reform and narrow.
I wholeheartedly agree, but there is no appetite for serious tax increases among Republicans or serious spending cuts among Democrats. Pretty sure we're going to need either inflation/bond markets to enforce fiscal discipline on the US.
Well, I think we’ll have to agree to disagree on an expanding set of considerations, notably the narrative.
The data sharing that he's referring to mostly have to do with restrictions when working with individual-level data that must be analyzed within a federal research data center. The number of statistical agencies isn't really a problem unless you think BLS should start compiling crime data.
Your comment would make sense if Muskaswamy actually cared about making government more efficient, rather than wanting to break it for teh lulz.
I’m trying to withhold complete cynicism that there’s anything going to happen with DOGE other than headline-grabbing “waste”, rather than promoting actual best practices for effective government.
Economic statistics get revised for two reasons:
1) When they move in a direction I don't like, it's because my political enemies are perpetrating a fraud to cover their incompetence.
2) When they move in a direction I do like, it's because a fraud perpetrated by my political enemies has been exposed and the record is being set straight.
This nerdery is why we subscribe. Related: any of you who haven't read Jennifer Pahlka's _Recoding America_ should.
Or her new state capacity report!
https://www.niskanencenter.org/policy/state-capacity/
Also check out the great discussion of it on the Statecraft podcast/substack!
I have great trust in our government statistical agencies. They are incredibly professional.* I have no trust in the Trump administration when it comes to keeping its hands off them when/if the numbers start turning sour.
Jed notes that political interference is one of the risks to data integrity. I hope in his next monthly post he updates us on what level of risk we're starting to face from Trump appointees, and continues to report on that as the administration unfolds.
*FRED is such a treasure that perhaps it should be considered for the economics Nobel? It has done more for broad-based understanding of the economy than any number of recent prize winners.
Right, there are things that could be better on the margin with our statistical agencies (many of these things would require Congress to act fwiw), but honestly we should all be proud of the data our government makes publicly available. These agencies (and sub-agencies) a great example of competent government doing things that are tremendously valuable that most people don't think about that often.
Michelle Obama said "when they go low, you go high." I feel like the theme of this blog lately has been "when they go crazy, you go normal."
Ah, famous last words.
She should have said “They bring a knife, you bring a gun.”
Maybe normal is a gun in this situation?
Informative article. However, while the author worries about "Inappropriate political interference," how we currently measure race and ethnicity (perhaps among other measures) is inherently political. The current categories are somewhat incoherent, mixing skin color (white, black) with geographic origins (Middle East and North Africa). Political interest groups (mainly progressive) have an outsized impact on its measurement.
Race and ethnicity don't have coherent definitions, and vary within cultural context. Different parts of the world define what is/isn't in these categories and how they interact with one another differently. Creating an 'objective' definition isn't possible. And yet, given the realities of the world, it would be a total abdication of responsibility to simply fail to measure these constructs. Not being able to differentiate African Americans from Whites in your data set isn't acceptable because it's very hard to create a conceptually satisfying framework.
Additionally, because of the long time-series feature of government statistics and their use by third parties (like me!) to benchmark and use for analysis, the government is shackled by an immense weight of history. The announced update to the OMB standards - collapsing Hispanic from an ethnicity into a race, and including MENA as a race separate from White - is the first update since 1997.
I'm not sure who you think defines these questions, but US statistical agencies take this stuff seriously. They do a ton of research and validation testing. And I'm not sure which progressive interest groups you think are advocating for changing things once every 30 years, but I don't know them.
“…would be a total abdication of responsibility to simply fail to measure these constructs”
Not using arbitrary constructs is an abdication of responsibility? You can’t be serious.
When that construct is critical to understanding the population you're attempting to measure? You do your best.
Headcount. Age. Sex. Number of people in the household and relatedness. Type of dwelling. Own or rent. Citizenship. Immigration status. Employment status.
Only headcount is critical.
Critical for what?
For the constitutional purpose of the census.
Yes, US statistical agencies take this seriously, and they do considerable testing, but the fact is that advocacy groups have significant influence on the process. Most changes made in racial classification increase the count of members of advocacy groups (which inevitably reduce the count of the white population). Here's one book that discusses some of these issues, from a fairly sympathetic viewpoint: https://journals.sagepub.com/doi/full/10.1177/0002716218756629
Other critics are not so kind.
Adding the MENA category seems like an advocacy group driven change. As a long time Census Bureau data user, it makes no sense to me.
An interesting read, and definitely merits more time than I can give it now. I would just note that advocates can make bad arguments for good outcomes, and considering the socially constructed and ever-shifting nature of these constructs, sometimes the experts are going to agree through other sources of knowing.
Maybe this is naive, and perhaps we're already doing it, but when an objective measurement isn't possible, one solution is to support multiple measurements.
One wonders what appropriate political interference looks like.
It looks like Democrats doing it
It looks like politics. Nasty.
One thing I don't understand is that we're just fine with outmoded surveys and revisions up to a year later. We have the data itself across agencies to track this stuff in near real time if we wanted to. The entire private sector world has been investing into CDPs that organize their customer data across online and offline touchpoints, in real time, for the use of ads ROI for over a decade now. Working with general survey data like this is a policy choice to be inaccurate in the age of immediately available data.
My arg for a public CDP to resolve these self-inflicted data issues -- https://open.substack.com/pub/socialdawn/p/the-us-needs-a-public-cdp?utm_source=share&utm_medium=android&r=9xf6a
Yep. I’m a fan of making everyone participate in these things fully. Get a new job, report it to the state. Get a raise, report it to the state. For W2 employees, the Work Number aggregates this info anyway! I’ve only had one W2 job in my whole life that isn’t on there, and I only worked there 3 months. Even one of my 1099 jobs is on there!
I hope at the very least our compilers of official statistics have data as solid as the Work Number’s.
I read your post linked below, and I think it is worth thinking seriously about differences between how the private sector uses and collects data versus the public sector. There are technological limitations in the public sector, but there also legal limitations that prevent how federal agencies can collect or use data. Frankly, the private sector is much less constrained than the public sector, and a lot of the "modernizing" required would need Congress to update laws like the Privacy Act, as one example.
These limitations are policy choices made without an iota of provable security benefits. The constraints are self-imposed and would not require congress to update any laws at all in order to start building CDP-like capabilities out of the reams of non-PII digital data available. E.g. See the slow rollout of login.gov, that should simply be mandated to use across the board for every agency that touches constituents.
The sheer amount of granular interactivity data held within the GSAs DAP program (https://analytics.usa.gov/) most Fortune 500 companies would die for, and it's mostly gone totally unused for work outside basic annual program analysis of user counts and volume
I think we agree, but might be talking about slightly different things or focusing on different components For some things that rely surveys, for example, it's not clear to me what alternatives exist for collecting individual-level data or in real-time. For example, the Survey of Consumer Finances (https://www.federalreserve.gov/econres/scfindex.htm) collects information from households that couldn't be collected otherwise.
I think this is different from a question of how the government collects and uses data more broadly. The government does not provide a great customer experience. Basically, both these are true:
1. The government does not manage or use the data it collects effectively
2. The government faces limitations that make it different from the private sector in what it can and cannot do
Sure, we do need legislation for specific survey data collection regimes, I agree with that. My broader point is that the survey of consumer finances is probably less accurate than a model of consumer finances based on inputs that a CDP could control -- macro data, user applications to specific gov benefits programs, more detail into bank data they are required to submit, etc. There are different ways to measure these things and the over reliance on surveys opens up all of these quantitative issues that, measured via other means, would be far less of a concern.
Wouldn’t relying on job postings result in a significant overestimate of available jobs? It costs a company nothing to post a position they may not really intend to fill.
Ok…but where do the Duke brothers and my FCOJ futures come in?
I feel like this is core Slow Boring content, a post that cuts against our hyperbolic and conspiratorial age.
Make sure this post stays paywalled so JD Vance and Xi Jinping don’t discover the dark technology of upward population controls
The government cooked the books in the 2020 census with group quarters imputation. No reason to think they aren't doing similar things elsewhere.
Aaron, you are a respected data analyst and a long time user of Census data. I'm very curious as to why you see the group quarters imputation in response to disruption of normal procedures by the pandemic as "cooking the books". (Context is that they improvised an unplanned procedure in 2020 because due to the pandemic, they had group quarters that showed as occupied in January/February 2020 in their Group Quarters Advanced Contact, but provided no data or inconsistent data during July-October follow ups). This affected 2% of all Group Quarters nationwide. The American Statistical Association independent assessment of Census data quality in August 2021 found GQ imputation as adding minimal risk at the state level, though local risk would be higher. The Bureau published a detailed report in 2023 of their methodology. https://www2.census.gov/library/publications/decennial/2020/2020-census-group-quarters-imputation.pdf You can disagree with the approach, but why would you call it "cooking the books:"?
Also how could they ever cook the books with this approach? It's just 2% of GQ and aren't GQ < 1% of all residences so just multiply it out.
Could you explain this?
It's a conspiracy theory. Here's the pubic results of the analysis: https://www2.census.gov/library/publications/decennial/2020/2020-census-group-quarters-imputation.pdf
All the data has pubic results if that’s your fetish.
Yes, they actually did group quarters imputation which is inherently illegitimate.
He can explain it but without smoking crack, you likely won't find it convincing (or you can google "2020 group quarters imputation conspiracy"). \S
> to subtract undocumented immigrants from state Congressional apportionment counts
Why is this thought to undermine integrity of official statistics?
Congressional apportionment is based on the number of persons in a state, including persons who cannot legally vote such as minors, legal permanent residents, felons who have had their voting rights stripped, etc. I’ve seen people argue that apportionment should be based only on the number of registered voters or only on the voting-age (and voting-eligible) population but under current law that’s not the way things have been set up.
i mean the constitution pretty clearly just says "free persons" (excluding Indians as Ken points out) so excluding children or non-citizens would require a constitutional amendment.
That doesn’t bear on the statistics, though.
The reason it damages official statistics is that controversies around immigrants and the Census make it harder for Census to get immigrants to respond to surveys. Source: getting immigrants to respond to surveys was my job at one time.
…excluding Indians not taxed.
I don't disagree with anything here but I feel like I'm missing context -- like this is a response to some kind of conspiracy theory I'm unaware of.
Basically every time revisions happen, some Republican with a huge social media following cries foul.
It's here as a thing to be linked to from social media when folks who don't understand how and why there are revisions to official statistics get freaked out when they find out that there was a revision to an official statistic.
Because this’ll show ‘em! /s
I think the fear is Trump will start cooking official statistics while claiming he is correcting past impropriety, as he did with election administration.
https://paulkrugman.substack.com/p/the-real-threat-of-fake-numbers?utm_campaign=posts-open-in-app&triedRedirect=true
Let's not forget Richard Nixon's hunt for Jews in the BLS in 1971.
Every once in a while on twitter: https://www.noahpinion.blog/p/in-which-david-sacks-and-balaji-raise?triedRedirect=true
https://x.com/DavidSacks/status/1665090176601956352
https://x.com/balajis/status/1665007599224516609
(1) David Sacks sucks but (2) that Twitter community notes thing works pretty well here. Really simple explanation why Sacks' statement is wrong.
Huh, but what Sacks said is very much true. Both the revision and the original can't both be true.
Is the issue is that in context that is supposed to be read as a problem? Or was there some other tweet in the thread I missed?
My understanding is there's at least four moving parts in every report: (1) payroll survey, (2) household survey, (3) household adjustments, (4) seasonal adjustments and Sacks misunderstood which was what.
Here's the pinned reply answer Sacks: https://x.com/josephpolitano/status/1665094829477441539?s=46&t=pbja_JVdJa2yFAIKJipKZQ
EDIT: I don't really care about the specific tweet either. See (1). My point was just that those new community notes seem to be working pretty well.
Ohh I see I didn't realize the first link was supposed to be about sacks too.
The sad thing is that any econ grad student could tell the tech guys here why they're wrong, but the tech guys aren't willing to rise to the level of competence of a mediocre grad school.