"No fun predictions" is a bad call! The reason to do fun ones is to keep yourself interested in the exercise and make it less painful to sit down and do. If you're worried about dragging down your calibration score, grade them separately.
One thing I realized reading this list is the challenge of calibration when the predictions are correlated. You got one major thing wrong, inflation, and it messed up a lot of questions. That, sort of by necessity means either you’re right or wrong about a lot of questions so you’ll look too over/under confident each year.
A reason to include “fun” questions is they tend to be uncorrelated and so help you track calibration better.
As one of Dr. Tetlock's Superforecasters®, I have to give you an A+ for your process. Here's a list of things you did that most people fail to do:
Made a lot of forecasts. (I'd recommend doing 100, but we all have time constraints)
Wrote your forecasts down.
The forecasts had clear resolution criteria.
You shared the list.
Attached a probability to each forecast. (This is huge.)
Revisited the results, especially the ones in which you were over-confident. (Essentially everyone is over-confident, but good forecasters are less so.)
Analyzed the results and learned from errors.
Wrote up a new list and did it again.
Of course, you can improve your scores by being part of group of other smart people who you look to for reasons to change your mind, but as a sole forecaster, I'd say you've got a lot of promise, kid.
Regarding running, don't get discouraged! You don't compare yourself to the other runners you see (there will always be someone faster anyway). You compare yourself to all the people sitting on their couches at home who you don't see.
I used to be pretty overweight when I started running regularly in my mid-to-late 20s, but it put me in a better shape than I ever was when I was younger. I will never judge someone for getting out and moving at whatever pace works for them.
If Stephen Breyer doesn’t retire in 2022 then he might as well join the federalist society. He will certainly be no “liberal” if he’s so willing to place his ego over protecting his legacy.
1. I think your accounting of your past predictions is extraordinarily useful. Most professional predictors never perform this basic exercise, and I'd say your record is considerably better than most.
2. I took up running from nada quite a while ago; unexpectedly I stuck with it and found that my stamina/endurance/speed continued to improve over time. Good luck!
If you follow Michael Pettis you can get a sense of why China is so recalcitrant, they simply don't have the money. They see a very big crisis coming that they don't have a good solution for, and importing American goods would only exacerbate that crisis.
2022 is going to be a very important year for the future of China. They've got to make a decision about how they're going to address their housing/GDP/household income crisis and "build more highways, but this time in Africa" isn't a viable solution, as even Xi is now publicly admitting.
China is going to become more vulnerable to tariffs and international pressure, as the last thing they can afford is any slowdown in exports. It may not seem like it right now, but China is in a very vulnerable position.
Dec 30, 2021·edited Dec 30, 2021Liked by Milan Singh
As a retired intel analyst, this is near and dear to my heart. Forecasting and prediction is, indeed, a very humbling activity made all the more difficult because of human cognition, our reliance on pattern matching and confirmation bias, and our tendency to assume we know more than we actually do. And one very important thing to consider is that accurate predictions (as opposed to lucky guesses) are very often not possible because so many things are emergent or driven by emergent factors.
In the intelligence world the most obvious example of this is estimating what other countries will do in the future. This is nigh impossible in the many cases when the countries in question haven't actually decided what they intend to do.
Anyway, before I descent into the weeds and start quoting Sherman Kent and Cynthia Grabo, I'll just say I appreciate this exercise. If nothing else, it's a useful tool to recalibrate one's ability for introspection.
I do have a couple of recommendations, though, if you want to up the rigor of your predictions:
First, I'd ditch the percentages which imply a false precision. I would instead adopt the estimative language ("words of estimative probability) similar to what the intelligence community or the IPCC uses. Or just adopt theirs.
Secondly, in addition to making predictions regarding the likelihood of something occurring, consider also providing the confidence level for your prediction. This is also standard practice in the IC, IPCC and others.
Third, and this is the time-consuming part, is to show your work and explain your reasoning, particularly when it comes to the confidence level. This is a good method for checking the validity of your assumptions and it will also let you know when you made a correct prediction, but for the wrong reasons.
So for example, in your first prediction, you might change it to something like this:
"1. Democrats lose both houses of Congress (virtually certain, high confidence). Historical patterns, current trends and seat vulnerability analysis all strongly support Democrats losing both houses of Congress."
And if you want to up the game even further, you can assess what indicators or factors would need to change for your prediction to be wrong. So in the case of Congress one might add:
"Democrats retaining control of one or both houses of Congress would likely require a fundamental shift in current political and economic trends as well as circumstances (such as an emergent crisis) that fundamentally shifts the political landscape prior to the election (low confidence)."
Now, all of that is a lot of work, but it can be a useful exercise to do on a few predictions. And, down the line, this is very helpful to determine why a prediction was right or wrong - and that, IMO, is more important than scoring for these types of predictions.
I remember back in August the comments section predicted that the fall of Afghanistan wouldn’t matter in 3 months. Now google search results shows that is the case. 100m searches for Afghan in august to like 3 million now.
“The same party wins both Senate races in Georgia (95%) YES
This one was obvious.“
Was it? The actual results make 95% seem overconfident to me. Some very rough rule of thumb reasoning here: the two races ended up having a 0.8% difference in margin, and I expect the vast majority of possible outcomes for the race end up in a band that’s smaller than (20 * 0.8 = 16%), which leads me to think the chance of landing in that 0.8% range where the races split is higher than 5% (since the split outcomes are closer to the centre of that band).
Love this—thanks for writing it. One of the countless things I don’t understand about the world right now is the certainty so many people seem to feel about so many things. People with a platform to express such certainty publicly rarely seem daunted by (or possibly even aware of) the actual track record of their proclamations.
I think adding the going rate from metaculus, predictid, etc... as a point of comparison would be massively useful in evaluating your success next year. If the market says 99% but you say 60% it is in some ways very impressive for you should the event not happen.
I recommend lifting, which is less hard to get into than is it commonly portrayed, and is also not just for dudes who want to get huge and impress other dudes. If you frequent places where men people talk about lifting, they will talk almost exclusively about programs involving barbells, and there are a couple popular beginner barbell programs, but those obviously require access to that equipment. Nerd Fitness has a beginner bodyweight program (no equipment) and a beginner kettlebell program and they are both fine. R/Fitness has a much more intensive beginner bodyweight program in their wiki as well.
Dang these new predictions are much more depressing ones. Hopefully your poor track record from last year repeats itself and we have a better than expected 2022!
Predictions are hard
"No fun predictions" is a bad call! The reason to do fun ones is to keep yourself interested in the exercise and make it less painful to sit down and do. If you're worried about dragging down your calibration score, grade them separately.
One thing I realized reading this list is the challenge of calibration when the predictions are correlated. You got one major thing wrong, inflation, and it messed up a lot of questions. That, sort of by necessity means either you’re right or wrong about a lot of questions so you’ll look too over/under confident each year.
A reason to include “fun” questions is they tend to be uncorrelated and so help you track calibration better.
As one of Dr. Tetlock's Superforecasters®, I have to give you an A+ for your process. Here's a list of things you did that most people fail to do:
Made a lot of forecasts. (I'd recommend doing 100, but we all have time constraints)
Wrote your forecasts down.
The forecasts had clear resolution criteria.
You shared the list.
Attached a probability to each forecast. (This is huge.)
Revisited the results, especially the ones in which you were over-confident. (Essentially everyone is over-confident, but good forecasters are less so.)
Analyzed the results and learned from errors.
Wrote up a new list and did it again.
Of course, you can improve your scores by being part of group of other smart people who you look to for reasons to change your mind, but as a sole forecaster, I'd say you've got a lot of promise, kid.
1. Matt Y will start podcasting again in 2022 (and not just as a guest). 95%
Regarding running, don't get discouraged! You don't compare yourself to the other runners you see (there will always be someone faster anyway). You compare yourself to all the people sitting on their couches at home who you don't see.
I used to be pretty overweight when I started running regularly in my mid-to-late 20s, but it put me in a better shape than I ever was when I was younger. I will never judge someone for getting out and moving at whatever pace works for them.
If Stephen Breyer doesn’t retire in 2022 then he might as well join the federalist society. He will certainly be no “liberal” if he’s so willing to place his ego over protecting his legacy.
1. I think your accounting of your past predictions is extraordinarily useful. Most professional predictors never perform this basic exercise, and I'd say your record is considerably better than most.
2. I took up running from nada quite a while ago; unexpectedly I stuck with it and found that my stamina/endurance/speed continued to improve over time. Good luck!
If you follow Michael Pettis you can get a sense of why China is so recalcitrant, they simply don't have the money. They see a very big crisis coming that they don't have a good solution for, and importing American goods would only exacerbate that crisis.
2022 is going to be a very important year for the future of China. They've got to make a decision about how they're going to address their housing/GDP/household income crisis and "build more highways, but this time in Africa" isn't a viable solution, as even Xi is now publicly admitting.
China is going to become more vulnerable to tariffs and international pressure, as the last thing they can afford is any slowdown in exports. It may not seem like it right now, but China is in a very vulnerable position.
As a retired intel analyst, this is near and dear to my heart. Forecasting and prediction is, indeed, a very humbling activity made all the more difficult because of human cognition, our reliance on pattern matching and confirmation bias, and our tendency to assume we know more than we actually do. And one very important thing to consider is that accurate predictions (as opposed to lucky guesses) are very often not possible because so many things are emergent or driven by emergent factors.
In the intelligence world the most obvious example of this is estimating what other countries will do in the future. This is nigh impossible in the many cases when the countries in question haven't actually decided what they intend to do.
Anyway, before I descent into the weeds and start quoting Sherman Kent and Cynthia Grabo, I'll just say I appreciate this exercise. If nothing else, it's a useful tool to recalibrate one's ability for introspection.
I do have a couple of recommendations, though, if you want to up the rigor of your predictions:
First, I'd ditch the percentages which imply a false precision. I would instead adopt the estimative language ("words of estimative probability) similar to what the intelligence community or the IPCC uses. Or just adopt theirs.
Secondly, in addition to making predictions regarding the likelihood of something occurring, consider also providing the confidence level for your prediction. This is also standard practice in the IC, IPCC and others.
Third, and this is the time-consuming part, is to show your work and explain your reasoning, particularly when it comes to the confidence level. This is a good method for checking the validity of your assumptions and it will also let you know when you made a correct prediction, but for the wrong reasons.
So for example, in your first prediction, you might change it to something like this:
"1. Democrats lose both houses of Congress (virtually certain, high confidence). Historical patterns, current trends and seat vulnerability analysis all strongly support Democrats losing both houses of Congress."
And if you want to up the game even further, you can assess what indicators or factors would need to change for your prediction to be wrong. So in the case of Congress one might add:
"Democrats retaining control of one or both houses of Congress would likely require a fundamental shift in current political and economic trends as well as circumstances (such as an emergent crisis) that fundamentally shifts the political landscape prior to the election (low confidence)."
Now, all of that is a lot of work, but it can be a useful exercise to do on a few predictions. And, down the line, this is very helpful to determine why a prediction was right or wrong - and that, IMO, is more important than scoring for these types of predictions.
I remember back in August the comments section predicted that the fall of Afghanistan wouldn’t matter in 3 months. Now google search results shows that is the case. 100m searches for Afghan in august to like 3 million now.
“The same party wins both Senate races in Georgia (95%) YES
This one was obvious.“
Was it? The actual results make 95% seem overconfident to me. Some very rough rule of thumb reasoning here: the two races ended up having a 0.8% difference in margin, and I expect the vast majority of possible outcomes for the race end up in a band that’s smaller than (20 * 0.8 = 16%), which leads me to think the chance of landing in that 0.8% range where the races split is higher than 5% (since the split outcomes are closer to the centre of that band).
You’ve got through the crappy part of running! And there’s plenty of benefits to not taking it too seriously… namely not getting hurt
Happy New Year sir, a hurt runner
Love this—thanks for writing it. One of the countless things I don’t understand about the world right now is the certainty so many people seem to feel about so many things. People with a platform to express such certainty publicly rarely seem daunted by (or possibly even aware of) the actual track record of their proclamations.
I do, however, wish there were more fun ones!
I think adding the going rate from metaculus, predictid, etc... as a point of comparison would be massively useful in evaluating your success next year. If the market says 99% but you say 60% it is in some ways very impressive for you should the event not happen.
I recommend lifting, which is less hard to get into than is it commonly portrayed, and is also not just for dudes who want to get huge and impress other dudes. If you frequent places where men people talk about lifting, they will talk almost exclusively about programs involving barbells, and there are a couple popular beginner barbell programs, but those obviously require access to that equipment. Nerd Fitness has a beginner bodyweight program (no equipment) and a beginner kettlebell program and they are both fine. R/Fitness has a much more intensive beginner bodyweight program in their wiki as well.
Dang these new predictions are much more depressing ones. Hopefully your poor track record from last year repeats itself and we have a better than expected 2022!