12. and 22. don't seem like they should be the same given there were more deaths in 2021 than 2022. Also, No recession in 2021 should probably read 2022. I also think some of these were revised because I originally remember the inflation being lower than 6% at 80% and higher than 5% at 80% (for a 64% cumulative probably of the inflation rate being between 5 and 6. This was far too confident in my mind, so not surprised to see it revised - but do feel like there needs to be a point where these are set in stone for calibration sake.
2. I think it's helpful to make lots of forecasts on random stuff that will get resolved within days or even hours - you can do that by playing on PredictIt, Good Judgment Project, Metaculus, etc. or just by stuff like privately putting probabilities on the most popular answers to online polls before clicking "show results". I think most people who are good at forecasting do something like this.
Overall I think there's a lot about forecasting that is more like "tricks of the trade" and "having a feel for what the numbers mean" than like "understanding the world." I think most people who are good at forecasting have done enough rapid-feedback practice to get good at that part of it.
I just wanted to congratulate Matt on committing to daily exercise (running) and by extension improving his physical and mental health! Rooting for you!
If accuracy is the goal above all, you'll probably drift towards making safe predictions that will bore your readers to tears. It's more interesting to read some unexpected predictions from time to time where either I can see your reasoning, or you're making predictions about topics I don't follow.
22 and 23 seem weird to me when considered together. 30% chance of <4, 30% chance of >6, implies 40% chance of 4 to 6 - I guess that makes sense but is that really your view?
I am running my first marathon next weekend. Part of marathon training, I’m told, is to “train slow”, focusing on endurance and not on speed. I am quite slow, but that’s okay because, we’ll, I’m focused on endurance and not on speed.
But the mental image of Matt breezing right past me is more than I can bear. As soon as this race is over, I’m focusing on speed. This aggression cannot stand, man.
Most people don’t even try the new hard thing because they want to avoid the uncomfortable feelings.
So, keep it up! I enjoy hearing these updates about your life.
Btw, I like to think I’m pretty fit and I also think running _sucks_ and doing 4 miles is actually legit hard.
Also, when I see people out there who maybe don’t look that fit but who you can tell are committed, it’s inspiring to _me_ to go again the next day.
So keep it up — for yourself and for the rest of us!
Ps if you’re not trail running in Rock Creek, I highly recommend it, it is easily 1000x better than running around on the side walks and made running tolerable for me.
I think one of the reasons you have done poorly is that you are overly optimistic about how quickly national governments can change course, particularly when the change involves other national governments.
Just want to encourage you on the running! I started a handful of years ago, sounds like a similar journey to you at first. Now I regularly run 10+ miles for fun and run half marathons a few times a year. I still consistently rank in the bottom half of my age group in races, and have never, not one single time, gotten a faster time than my wife. But, who gives a shit. It’s great exercise, and now that I’m “good” at it (as in, I can really enjoy it and don’t get really miserable and sore during or after), it’s also like a mini meditation time in my day that where I can relax and think.
Anyway, just wanted to offer encouragement. I’ve made a career out of doing stuff I’m good at, which makes it naturally enjoyable. I still think I’m weirdly more proud though of making a hobby out of something I was naturally bad at (and sticking with it until it became enjoyable).
Also, since you’ve got substack money, I recommend you getting super nice, super soft shoes (like Brooks Glycerine). I wish I would have sooner, much easier on your joints, which makes you want to run more.
Damn, those congressional predictions are surprising to me! Is it really so likely that the Democrats will lose both houses? The House of Reps I get, but with the Senate, I was under the impression that the 2022 map was quite favorable to Dems, as they’re mostly defending safe seats and the seats that are open are in swing states like PA and WI. Anyone have the counterpoint to this?
Also, I’d put Macron’s chance of winning the French election as a bit higher. His approval ratings are staggeringly high for a French president (the last one, for reference, left office with 11% approval) and they only seem to be improving, and no challenger is particularly convincing.
I'd be curious about your reasoning behind this as it seems massively overconfident to me - Orbán is head-to-head in the polls (where, given the election system he gerrymandered for himself, he would have to lose by 4% or so to break even in parliamentary seats), the polls have been fairly stable in the last year (despite a remarkably poor pandemic response), the opposition (which has been forced by the new election mechanism to unite into a single election pseudo-party) is crippled by huge internal ideological differences, mistrust, competing financial interests and lack of any real leadership position, Orbán has subverted pretty much all theoretically independent state institutions (including election authorities, campaign spending authorities etc) into his service, has a practically infinite budget to spend on campaigning while being able to prevent the opposition from getting non-trivial resources, has taken over the vast majority of Hungarian media, and has gotten away with small-scale organized election fraud so there isn't anything stopping him from doing larger-scale fraud this time if he deems it necessary.
As others have said, thanks for putting up your predictions! I would much prefer to read a writer who made bad predictions, holds themselves accountable for it and tries to improve, to one who doesn't even care about accuracy - I'd assume the latter is just using their public writing as social signaling and not actually expressing useful hypotheses about the world.
I was curious how prediction quality can be quantified. The stats as stated in the post are not so useful because they omit the number of predictions. With those added (hope this will end up somewhat readable):
pct # exp act Br+ Br- BrΣ
60 5 3 2 0.16 0.36 1.4
70 7 4.9 2 0.09 0.49 2.63
75 1 0.75 1 0.0625 0.5625 0.0625
80 6 4.8 4 0.04 0.64 1.44
90 2 1.8 1 0.01 0.81 0.82
95 4 3.8 4 0.0025 0.9025 0.01
(That is, predicted percentage / number of predictions / expected number of hits in case of perfect calibration / actual number of hits / Brier score for one success / Brier score for one failure / total Brier score. Flipped the one <50% prediction to make things simpler.)
So, out of 25 answers you should have gotten 18.05 right if your stated percentages were all perfect predictions; you actually got 14 right. More formally, the average Brier score (mean squared error, basically) is around 0.25 (where 0 would have been the perfect score and 1 perfectly bad). Here is a random ranking from a formal forecasting competition: http://aiimpacts.org/wp-content/uploads/2019/02/image9.png (note they use a different definition of Brier score where they skip a division by 2, so it's distributed from 0 to 2; so the Slow Boring predictions would be in their 0.55 bin).
While it's not really meaningful to compare forecasting scores without somehow accounting for the difficulty of the question being predicted, that would put this blog on par with a below-average professional forecaster - which does not sound bad at all for a first attempt.
It looks like Matt generally overestimated the chances that political things would happen. By my count Matt made 9 predictions last year about whether or not a political thing would happen, where a thing happening requires people in government to do something. 5 were about whether a policy change would occur and 4 about whether a personnel change would occur. He thought that 6 of the 9 things were more likely than not to happen, and in expectation that 4.8 of them would happen. In fact, only 1 happened, and it was the most routine one: a person who had been nominated for a cabinet position was confirmed for that position.
The 9 predictions:
(R) No federal tax increases are enacted (95%) YES
Biden administration unilaterally relieves some but not all student debt (80%) NO
United States rejoins JCPOA and Iran resumes compliance (80%) NO
Israel and Saudi Arabia establish official diplomatic relations (70%) NO
U.S. and China reach an agreement to lift Trump-era tariffs (70%) NO
Nancy Pelosi sets a definitive retirement schedule (60%) NO
A vacancy arises on the Supreme Court (70%) NO
(R) Joe Biden ends the year as president (95%) YES
(R) Lloyd Austin not confirmed as Defense Secretary (60%) NO
(Here (R) stands for "reverse", meaning that Matt phrased the statement as "thing will not happen" rather than as "thing will happen".)
12. and 22. don't seem like they should be the same given there were more deaths in 2021 than 2022. Also, No recession in 2021 should probably read 2022. I also think some of these were revised because I originally remember the inflation being lower than 6% at 80% and higher than 5% at 80% (for a 64% cumulative probably of the inflation rate being between 5 and 6. This was far too confident in my mind, so not surprised to see it revised - but do feel like there needs to be a point where these are set in stone for calibration sake.
I think it's great that you're doing this, and that you owned the lack of awesome results the first time around.
If you want to get better at forecasting, I would suggest finding a faster feedback loop than once a year. In particular:
1. I would definitely suggest doing formal calibration training if you haven't already (you can try https://www.openphilanthropy.org/calibration or do a workshop).
2. I think it's helpful to make lots of forecasts on random stuff that will get resolved within days or even hours - you can do that by playing on PredictIt, Good Judgment Project, Metaculus, etc. or just by stuff like privately putting probabilities on the most popular answers to online polls before clicking "show results". I think most people who are good at forecasting do something like this.
Overall I think there's a lot about forecasting that is more like "tricks of the trade" and "having a feel for what the numbers mean" than like "understanding the world." I think most people who are good at forecasting have done enough rapid-feedback practice to get good at that part of it.
I just wanted to congratulate Matt on committing to daily exercise (running) and by extension improving his physical and mental health! Rooting for you!
If accuracy is the goal above all, you'll probably drift towards making safe predictions that will bore your readers to tears. It's more interesting to read some unexpected predictions from time to time where either I can see your reasoning, or you're making predictions about topics I don't follow.
22 and 23 seem weird to me when considered together. 30% chance of <4, 30% chance of >6, implies 40% chance of 4 to 6 - I guess that makes sense but is that really your view?
I am running my first marathon next weekend. Part of marathon training, I’m told, is to “train slow”, focusing on endurance and not on speed. I am quite slow, but that’s okay because, we’ll, I’m focused on endurance and not on speed.
But the mental image of Matt breezing right past me is more than I can bear. As soon as this race is over, I’m focusing on speed. This aggression cannot stand, man.
Happy New Year!
Most people don’t even try the new hard thing because they want to avoid the uncomfortable feelings.
So, keep it up! I enjoy hearing these updates about your life.
Btw, I like to think I’m pretty fit and I also think running _sucks_ and doing 4 miles is actually legit hard.
Also, when I see people out there who maybe don’t look that fit but who you can tell are committed, it’s inspiring to _me_ to go again the next day.
So keep it up — for yourself and for the rest of us!
Ps if you’re not trail running in Rock Creek, I highly recommend it, it is easily 1000x better than running around on the side walks and made running tolerable for me.
I'd put the odds of #28 at 10%---and if any agreement is reached it will be a crappy one for consumers.
Why no prediction about how likely the Metaverse will transform our world into paradise?
I think one of the reasons you have done poorly is that you are overly optimistic about how quickly national governments can change course, particularly when the change involves other national governments.
Just want to encourage you on the running! I started a handful of years ago, sounds like a similar journey to you at first. Now I regularly run 10+ miles for fun and run half marathons a few times a year. I still consistently rank in the bottom half of my age group in races, and have never, not one single time, gotten a faster time than my wife. But, who gives a shit. It’s great exercise, and now that I’m “good” at it (as in, I can really enjoy it and don’t get really miserable and sore during or after), it’s also like a mini meditation time in my day that where I can relax and think.
Anyway, just wanted to offer encouragement. I’ve made a career out of doing stuff I’m good at, which makes it naturally enjoyable. I still think I’m weirdly more proud though of making a hobby out of something I was naturally bad at (and sticking with it until it became enjoyable).
Also, since you’ve got substack money, I recommend you getting super nice, super soft shoes (like Brooks Glycerine). I wish I would have sooner, much easier on your joints, which makes you want to run more.
can we get odds on the chances Raffensberger wins his primary?
Damn, those congressional predictions are surprising to me! Is it really so likely that the Democrats will lose both houses? The House of Reps I get, but with the Senate, I was under the impression that the 2022 map was quite favorable to Dems, as they’re mostly defending safe seats and the seats that are open are in swing states like PA and WI. Anyone have the counterpoint to this?
Also, I’d put Macron’s chance of winning the French election as a bit higher. His approval ratings are staggeringly high for a French president (the last one, for reference, left office with 11% approval) and they only seem to be improving, and no challenger is particularly convincing.
Why no predictions with less than 60% confidence? Learning how to occasionally nail those feels like an important element to being a better predictor.
"Viktor Orbán loses power in Hungary (60%)"
I'd be curious about your reasoning behind this as it seems massively overconfident to me - Orbán is head-to-head in the polls (where, given the election system he gerrymandered for himself, he would have to lose by 4% or so to break even in parliamentary seats), the polls have been fairly stable in the last year (despite a remarkably poor pandemic response), the opposition (which has been forced by the new election mechanism to unite into a single election pseudo-party) is crippled by huge internal ideological differences, mistrust, competing financial interests and lack of any real leadership position, Orbán has subverted pretty much all theoretically independent state institutions (including election authorities, campaign spending authorities etc) into his service, has a practically infinite budget to spend on campaigning while being able to prevent the opposition from getting non-trivial resources, has taken over the vast majority of Hungarian media, and has gotten away with small-scale organized election fraud so there isn't anything stopping him from doing larger-scale fraud this time if he deems it necessary.
As others have said, thanks for putting up your predictions! I would much prefer to read a writer who made bad predictions, holds themselves accountable for it and tries to improve, to one who doesn't even care about accuracy - I'd assume the latter is just using their public writing as social signaling and not actually expressing useful hypotheses about the world.
I was curious how prediction quality can be quantified. The stats as stated in the post are not so useful because they omit the number of predictions. With those added (hope this will end up somewhat readable):
pct # exp act Br+ Br- BrΣ
60 5 3 2 0.16 0.36 1.4
70 7 4.9 2 0.09 0.49 2.63
75 1 0.75 1 0.0625 0.5625 0.0625
80 6 4.8 4 0.04 0.64 1.44
90 2 1.8 1 0.01 0.81 0.82
95 4 3.8 4 0.0025 0.9025 0.01
(That is, predicted percentage / number of predictions / expected number of hits in case of perfect calibration / actual number of hits / Brier score for one success / Brier score for one failure / total Brier score. Flipped the one <50% prediction to make things simpler.)
So, out of 25 answers you should have gotten 18.05 right if your stated percentages were all perfect predictions; you actually got 14 right. More formally, the average Brier score (mean squared error, basically) is around 0.25 (where 0 would have been the perfect score and 1 perfectly bad). Here is a random ranking from a formal forecasting competition: http://aiimpacts.org/wp-content/uploads/2019/02/image9.png (note they use a different definition of Brier score where they skip a division by 2, so it's distributed from 0 to 2; so the Slow Boring predictions would be in their 0.55 bin).
While it's not really meaningful to compare forecasting scores without somehow accounting for the difficulty of the question being predicted, that would put this blog on par with a below-average professional forecaster - which does not sound bad at all for a first attempt.
It looks like Matt generally overestimated the chances that political things would happen. By my count Matt made 9 predictions last year about whether or not a political thing would happen, where a thing happening requires people in government to do something. 5 were about whether a policy change would occur and 4 about whether a personnel change would occur. He thought that 6 of the 9 things were more likely than not to happen, and in expectation that 4.8 of them would happen. In fact, only 1 happened, and it was the most routine one: a person who had been nominated for a cabinet position was confirmed for that position.
The 9 predictions:
(R) No federal tax increases are enacted (95%) YES
Biden administration unilaterally relieves some but not all student debt (80%) NO
United States rejoins JCPOA and Iran resumes compliance (80%) NO
Israel and Saudi Arabia establish official diplomatic relations (70%) NO
U.S. and China reach an agreement to lift Trump-era tariffs (70%) NO
Nancy Pelosi sets a definitive retirement schedule (60%) NO
A vacancy arises on the Supreme Court (70%) NO
(R) Joe Biden ends the year as president (95%) YES
(R) Lloyd Austin not confirmed as Defense Secretary (60%) NO
(Here (R) stands for "reverse", meaning that Matt phrased the statement as "thing will not happen" rather than as "thing will happen".)