I worked on the Federal Reserve's staff macro forecast for a decade, starting in July 2008. I have learned how to be a vey good forecaster. I am. I affirm many of the points you made. Let me suggest some baby steps into the craft:
1) start with your 'most likely' forecast, that is, the 50% or the median forecast. That's easier than thinking through the entire distribution. state the conditions on your forecast. good forecasts are ALWAYS conditional. tell the 'story.' go beyond the outcomes, tell us what gets us there.
2) when you are ready to move off the most likely, think through some 'upside' and 'downside' risks. then tell us are the risks tilted to the downside or the upside.
3) a100% agree, re-elevate your forecast (most likely, risks, conditions) regularly. then UPDATE YOUR FORECAST. key is paying attention to ALL the data available. watch for blind spots.
4) Superforecasters are often'non experts.' why? one reason is that experts tend to have blind spots. they want so badly for their models to be right that they miss signs they are wrong. Fed staff grappled with that after missing the warning signs of the housing bubble. it's hard.
5) talk with non-experts, go out in the world. my walks have taught me and conversations with non-economists since Covid arrived have taught me so much. actively seek it out.
6) finally, be humble, especially, when your forecast proves right. after a really big hit, a senior officer came by my office and told me, "do not gloat in our next meeting. it could have been luck." and he said with a smile, "we all know you nailed it. you don't have to say a word."
I was fortunate and learned from the very best. no one has a crystal ball, but some of them give damn good advice.
This is what I respect about 538. Say what you will about Nate Silver (I think he’s kind of an asshole) he makes a serious effort to make his models and predictions as accurate as possible. He’s not just trying to boost his preferred ideological position or something.
Funnily enough, the place you see this most clearly is the 538 sports models, which are often wrong (for instance they seriously underrated the Lakers in the last NBA playoffs), but it’s obviously not the case that they’re wrong because of some secret bias against the Lakers or something by the creators.
Matt, will you also be interested in writing down what were the biggest wrong-headed takes you did in past 20 years of your career and what you learned from them? Like this predictions thing that will also help with building some pundit accountability
"But a 15 percent chance of a global pandemic is really bad! We need people to be able to discuss moderately improbable bad events without sounding like the boy who cried wolf."
I've been thinking about this more or less constantly since reading The Fifth Risk. This conversation about how to talk about and address improbable but really awful outcomes is very important.
This is relevant to a topic I’ve raised here before: the validation of COVID models. I mean both the formal mathematical models and whatever lies behind media statements like things are going to get worse or better. More specifically, I’m thinking about the flurry of news about people travelling over Thanksgiving. It was a mess in that there was no precision about how much extra travel was being feared, whether that mount or more or less resulted and whether given the amount of travel there was, was there additional cases, serious cases and deaths.
Just looking at the aggregate data, I can’t see any “Thanksgiving effect.” The increase in cases before and after Thanksgiving look about the same to me and the relation of deaths to reported cases has not changed a lot. But I’m just a guy on the Internet. Why isn’t there a NYT story about how public health officials got Thanksgiving right or wrong and why. How did travel compare to predictions? Did more of those who travelled have negative tests before doing so than expected? And how id the incident affect the parameters of or the structure of the models that officials use?
In 2009 I saw Lana del Rey, who wasn't calling herself Lana del Rey yet, play in a small club in Manhattan, and as we left I turned to my wife and said, "She's going to be playing arenas someday." That was pretty much my last prediction about anything, and I'd hate to break my streak.
Specifically, I recommend joining the challenges sponsored by The Economist and Fareed Zakaria GPS. I won both in 2018!
Good Judgement Inc. annually asks the top performers to become “Professional Superforecasters”. I’ve been a “Super” for 6 years. Here are some of our public forecasts:
I just read the NYT article about substack, where it is claimed that you have bitterly broken with liberalism. That seems to me to be an overstatement. You've always examined the ideological shibboleths of liberalism skeptically as far as I can recall. I often disagree with you, however you are consistently interesting and with the money.
I think the possibility that the GA Senate races end with Perdue/Warnock as the winners is underrated! Clearly the races will be defined by partisan turnout but the margins could be razor-thin and Kelly Loeffler is a uniquely awful candidate (not to mention how appointed incumbents in general see less of an advantage than elected incumbents). Warnock also tends to perform slightly better in polls than Ossoff. I don't think there will be many Perdue/Warnock voters, but I think there could be some only-Warnock Democratic voters and only-Perdue Republican voters. Not many, but possibly enough. Since we're being accountable I would put my "the same party wins both GA Senate races" at 80%.
Here’s a possibly dumb question but I don’t know statistics very well so I’ll ask... how do you check yourself on predictions where your confidence levels aren’t the same? If you have 10 predictions at 60%, that’s easy, you described it. But if you have 2 at 85%, 3 at 70%, 2 at 50%... is there a way to to combine all of those or do you just need to wait until you have enough of a sample size at each confidence level?
"Twitter is full of bullshit" is an amazing sub-heading. But I view cable news punditry as the true home of the unaccountable crap prediction - tons of air time to fill, talking heads saying whatever they want, no one ever checks except rarely the Daily Show.
Twitter's it's own thing - it's performative group-on-group public fighting. The incentive isn't just to be wrong in a provocative way, it's to dunk on the other side.
Hi Matt, great to see you doing this. Here are some ideas based on Scott Alexander's experience:
* it's easier to check the calibration of your predictions if they're all multiples of 10. Most people can't distinguish between 21 levels of confidence anyway, so using numbers ending in 5 could simply make your work harder
* it will help check your calibration if all the probabilities given are 50 percent or higher. Your Lakers prediction could be rephrased as a prediction with 75 percent confidence that someone other than the Lakers will win
Superforecasting is a great book, and I agree more people should read it! Thanks again, and I look forward to seeing how your predictions turn out.
I think the thing that surprises me most in this list is how bullish you are on the restoration of the JCPOA.
What's your thinking on Iran's incentives to rejoin at this stage? I have to imagine their internal pressures from the hardliners but also from just the general populace after the dual assassinations this year will make that difficult politically even if the leadership was so inclined, which also does not seem clear.
I agree that most of the discussion around the attempted "coup" has been awful and hysterical (much like everything with Twitter) and I know this wasn't the main point of your post, but there were a number of prominent writers (Zeynep Tufecki in particular) to point out that Trump was:
a) Intentionally disrupting the mail to make it harder/more confusing to mail in ballots in states like PA
b) Having local Republicans insist that mail-in ballots be counted AFTER Election Day, thus making it seem like he was ahead (again, in PA)
c) Leaning on judges he appointed to stop the counting of mail-in ballots
And the election was pretty close in the Electoral College! 45k votes in GA, WI, and AZ and Trump wins! The point was that, had it been one state (or even two), Trump might have found more sympathetic judges to flip the count for him. But with so many states, it would've been politically untenable, and the smart judges knew they couldn't do it credibly. I think it's more than fair to say those fears were NOT overblown, and had Trump been successful, it would've constituted a coup (albeit not a military coup).
Obviously what Trump is doing is bad, and what other Republicans are doing/not doing is bad (and probably worse), but I've been against the "coup" description for several reasons. First, asking Courts to do something, which is most of what has actually occurred (rather than merely been discussed), doesn't seem to be what is usually meant by the word coup, but more important has been what is implied: no-one who has called this a coup really seems to acting like it is one.
If this was a coup, what would that imply people who oppose him and it should do? Should there be a counter-coup? Is violence not just an available option, and justified, but wouldn't a coup be so bad that perhaps failing to engage in some level of violence in response to it is actually bad? If there's a coup happening, why is the only response to it tweeting and op-ed-ing instead of rounding up coup supporters to stop the overthrow of democracy? etc.
I am happy to say I don't think this is a coup, and don't think violence is justified to stop it, but how do people who think this is a coup justify taking the same actions in response to it that I am (waiting for the courts to reject idiotic claims, waiting for the electors to vote, waiting for Congress to count the ballots, and waiting for inauguration)?
I worked on the Federal Reserve's staff macro forecast for a decade, starting in July 2008. I have learned how to be a vey good forecaster. I am. I affirm many of the points you made. Let me suggest some baby steps into the craft:
1) start with your 'most likely' forecast, that is, the 50% or the median forecast. That's easier than thinking through the entire distribution. state the conditions on your forecast. good forecasts are ALWAYS conditional. tell the 'story.' go beyond the outcomes, tell us what gets us there.
2) when you are ready to move off the most likely, think through some 'upside' and 'downside' risks. then tell us are the risks tilted to the downside or the upside.
3) a100% agree, re-elevate your forecast (most likely, risks, conditions) regularly. then UPDATE YOUR FORECAST. key is paying attention to ALL the data available. watch for blind spots.
4) Superforecasters are often'non experts.' why? one reason is that experts tend to have blind spots. they want so badly for their models to be right that they miss signs they are wrong. Fed staff grappled with that after missing the warning signs of the housing bubble. it's hard.
5) talk with non-experts, go out in the world. my walks have taught me and conversations with non-economists since Covid arrived have taught me so much. actively seek it out.
6) finally, be humble, especially, when your forecast proves right. after a really big hit, a senior officer came by my office and told me, "do not gloat in our next meeting. it could have been luck." and he said with a smile, "we all know you nailed it. you don't have to say a word."
I was fortunate and learned from the very best. no one has a crystal ball, but some of them give damn good advice.
Scott Alexander has been doing this since 2014 I think? See if you can outscore him!
This is what I respect about 538. Say what you will about Nate Silver (I think he’s kind of an asshole) he makes a serious effort to make his models and predictions as accurate as possible. He’s not just trying to boost his preferred ideological position or something.
Funnily enough, the place you see this most clearly is the 538 sports models, which are often wrong (for instance they seriously underrated the Lakers in the last NBA playoffs), but it’s obviously not the case that they’re wrong because of some secret bias against the Lakers or something by the creators.
Matt, will you also be interested in writing down what were the biggest wrong-headed takes you did in past 20 years of your career and what you learned from them? Like this predictions thing that will also help with building some pundit accountability
"But a 15 percent chance of a global pandemic is really bad! We need people to be able to discuss moderately improbable bad events without sounding like the boy who cried wolf."
I've been thinking about this more or less constantly since reading The Fifth Risk. This conversation about how to talk about and address improbable but really awful outcomes is very important.
This is relevant to a topic I’ve raised here before: the validation of COVID models. I mean both the formal mathematical models and whatever lies behind media statements like things are going to get worse or better. More specifically, I’m thinking about the flurry of news about people travelling over Thanksgiving. It was a mess in that there was no precision about how much extra travel was being feared, whether that mount or more or less resulted and whether given the amount of travel there was, was there additional cases, serious cases and deaths.
Just looking at the aggregate data, I can’t see any “Thanksgiving effect.” The increase in cases before and after Thanksgiving look about the same to me and the relation of deaths to reported cases has not changed a lot. But I’m just a guy on the Internet. Why isn’t there a NYT story about how public health officials got Thanksgiving right or wrong and why. How did travel compare to predictions? Did more of those who travelled have negative tests before doing so than expected? And how id the incident affect the parameters of or the structure of the models that officials use?
In 2009 I saw Lana del Rey, who wasn't calling herself Lana del Rey yet, play in a small club in Manhattan, and as we left I turned to my wife and said, "She's going to be playing arenas someday." That was pretty much my last prediction about anything, and I'd hate to break my streak.
For those interested in forecasting, I recommend signing up for the commercial offshoot of Tetlock’s work:
https://www.gjopen.com/
Specifically, I recommend joining the challenges sponsored by The Economist and Fareed Zakaria GPS. I won both in 2018!
Good Judgement Inc. annually asks the top performers to become “Professional Superforecasters”. I’ve been a “Super” for 6 years. Here are some of our public forecasts:
https://goodjudgment.io/superforecasts/#1418
I just read the NYT article about substack, where it is claimed that you have bitterly broken with liberalism. That seems to me to be an overstatement. You've always examined the ideological shibboleths of liberalism skeptically as far as I can recall. I often disagree with you, however you are consistently interesting and with the money.
I think the possibility that the GA Senate races end with Perdue/Warnock as the winners is underrated! Clearly the races will be defined by partisan turnout but the margins could be razor-thin and Kelly Loeffler is a uniquely awful candidate (not to mention how appointed incumbents in general see less of an advantage than elected incumbents). Warnock also tends to perform slightly better in polls than Ossoff. I don't think there will be many Perdue/Warnock voters, but I think there could be some only-Warnock Democratic voters and only-Perdue Republican voters. Not many, but possibly enough. Since we're being accountable I would put my "the same party wins both GA Senate races" at 80%.
Here’s a possibly dumb question but I don’t know statistics very well so I’ll ask... how do you check yourself on predictions where your confidence levels aren’t the same? If you have 10 predictions at 60%, that’s easy, you described it. But if you have 2 at 85%, 3 at 70%, 2 at 50%... is there a way to to combine all of those or do you just need to wait until you have enough of a sample size at each confidence level?
"Twitter is full of bullshit" is an amazing sub-heading. But I view cable news punditry as the true home of the unaccountable crap prediction - tons of air time to fill, talking heads saying whatever they want, no one ever checks except rarely the Daily Show.
Twitter's it's own thing - it's performative group-on-group public fighting. The incentive isn't just to be wrong in a provocative way, it's to dunk on the other side.
Hi Matt, great to see you doing this. Here are some ideas based on Scott Alexander's experience:
* it's easier to check the calibration of your predictions if they're all multiples of 10. Most people can't distinguish between 21 levels of confidence anyway, so using numbers ending in 5 could simply make your work harder
* it will help check your calibration if all the probabilities given are 50 percent or higher. Your Lakers prediction could be rephrased as a prediction with 75 percent confidence that someone other than the Lakers will win
Superforecasting is a great book, and I agree more people should read it! Thanks again, and I look forward to seeing how your predictions turn out.
I think the thing that surprises me most in this list is how bullish you are on the restoration of the JCPOA.
What's your thinking on Iran's incentives to rejoin at this stage? I have to imagine their internal pressures from the hardliners but also from just the general populace after the dual assassinations this year will make that difficult politically even if the leadership was so inclined, which also does not seem clear.
I agree that most of the discussion around the attempted "coup" has been awful and hysterical (much like everything with Twitter) and I know this wasn't the main point of your post, but there were a number of prominent writers (Zeynep Tufecki in particular) to point out that Trump was:
a) Intentionally disrupting the mail to make it harder/more confusing to mail in ballots in states like PA
b) Having local Republicans insist that mail-in ballots be counted AFTER Election Day, thus making it seem like he was ahead (again, in PA)
c) Leaning on judges he appointed to stop the counting of mail-in ballots
And the election was pretty close in the Electoral College! 45k votes in GA, WI, and AZ and Trump wins! The point was that, had it been one state (or even two), Trump might have found more sympathetic judges to flip the count for him. But with so many states, it would've been politically untenable, and the smart judges knew they couldn't do it credibly. I think it's more than fair to say those fears were NOT overblown, and had Trump been successful, it would've constituted a coup (albeit not a military coup).
Obviously what Trump is doing is bad, and what other Republicans are doing/not doing is bad (and probably worse), but I've been against the "coup" description for several reasons. First, asking Courts to do something, which is most of what has actually occurred (rather than merely been discussed), doesn't seem to be what is usually meant by the word coup, but more important has been what is implied: no-one who has called this a coup really seems to acting like it is one.
If this was a coup, what would that imply people who oppose him and it should do? Should there be a counter-coup? Is violence not just an available option, and justified, but wouldn't a coup be so bad that perhaps failing to engage in some level of violence in response to it is actually bad? If there's a coup happening, why is the only response to it tweeting and op-ed-ing instead of rounding up coup supporters to stop the overthrow of democracy? etc.
I am happy to say I don't think this is a coup, and don't think violence is justified to stop it, but how do people who think this is a coup justify taking the same actions in response to it that I am (waiting for the courts to reject idiotic claims, waiting for the electors to vote, waiting for Congress to count the ballots, and waiting for inauguration)?