70 million is 1% of humanity, not 0.1%! Didn’t anyone else catch this???
7 million people, scattered across the globe and dealing with post-nuclear conditions, would be much more likely to die out over a few thousand years than 70 million.
Lots of skepticism for AI/AGI risk. To use an analogy, I think we're at the point where people first discovered some basic principles of nuclear physics and someone goes "whoa - there's a LOT of energy in here - if someone ever harnessed this energy, maybe for a weapon, it wouldn't be on the same playing field as what we have now with conventional chemistry."
And everyone in the comments section is like "y'all don't even have a weapon design on paper, let alone a prototype."
I think part of why we are seeing so many criticisms of EA right now in the incentives of the criticizers. EA people are more open to criticism than anyone else I know of and engage with criticism content, share and repost and reply to it. (See Astral Codex for more details).
If you lots of people respond in good faith to a certain type of content, read and engage with it, content creators will create a ton more of that type of content. If I wrote a blog post about the failure of EA people to wear pointy enough hats and it got read a million times, drove a ton of clicks on my blog and ad traffic, and dozens of people responded with point by point rebuttals and agreements, how many weeks before the Atlantic and NYT would have articles about EA's pointy hat problem?
We already know the long-term outcome here: the Butlerian Jihad, the banning of AI, the switch to a spice-based economy, the rise of Muad'dib, and eventually we get ruled by a giant god-worm.
I think that utilitarian minded people are just more likely to be people who like hard sci-fi and therefore they think more about AI and colonization then the average person. Or maybe reading sci-fi makes a person more amenable to utilitarianism. Someone should do a study.
I feel like MY slightly misrepresents what most people who consider themselves effective altruists think about AI risk. Though I certainly could be wrong about what the predominant position is, my understanding is that EA people aren't convinced that extinction risk is coming, but instead that AI is coming (most people agree on this) and even if there's only a .01% chance of it being a legitimate extinction risk, that very puts it in the range of things that are worth caring about. EX:
Malaria Prevention: $100 per life save= .01 lives per dollar
AI risk: p(extinction from AI= .0001), p(extinction from AI with additional $100Million)=.00001
Also, some EA people think about the additional risk not of sentient AIs destroying humans, but humans creating child or animal level intelligent sentient AIs of whom we provide no rights, and cause great suffering to. We could create billions, or trillions of these which introduce a ton more suffering into the world.
The one comment I'll make is that it's been...interesting...seeing just how EA-adjacent journalist and folks interact with the public in light of William MacAskill's book and media tour...
To be quite frank...the copy that's been written and the questions that have been asked have been a bit more...lukewarm at their warmest than I would have expected...and while the criticisms have not been as cutting as they could have been, they have certainly been there if you know what to look for.
It mostly seems like behind the scenes...there is a lot of dissent and disagreement about the decision to center Longermism so much and it sort of feels like a lot of folks suddenly have knots in their throats as to whether or not they should speak out more loudly about it...or just sort of sit back and hope the entire movement doesn't blow up because one of it's key figures stumbles over a particularly cutting question.
Without being present for a lot of those comments, it just gives me a sense that a lot of folks feel that key figures are playing with fire and going to ruin all the good faith the movement has earned up until this point in time....
On the other hand, waxing philosophical, if AI does wipe out most of humanity then it will clearly have done a service to the planet at large. Give other species a chance.
Well here's my two cents. Which I will put in an interest bearing expense account and when it reaches a billion dollars the future can use it to do whatever it wants. I am a longtermist. Not a very good one but hey, I never said I was.
If people are actual longtermists who worry about helping future humans, I might suggest the following way to spend their charitable dollars:
1. Put $100M into a fund that matures in 2122
2. If they can get 7% annual growth (long term stock market average), it will be worth $87B (although clearly there's a wide range of values based on actual return rate)
3. In 2122, Give Well can spend that money on whatever they think are the most urgent problems
4. Repeat every 10 years
The result will be spending $1B this century to provide several hundred billion dollars next century to help solve whatever urgent problems those humans have.
I'm sure there are lots of issues with this idea, but I think it might be nice to consider alongside the now standard EA causes.
Like many of you I was introduced to EA by StarSlateCodex, and I've found it impossible to see AI risk as anything but the Silicon Valley "grey tribe" version of BLM, which caused every moderately progressive organization to seize up and collapse under the weight of their own self-importance.
Nowhere is this more obvious than their bugaboo of choice - of course it's AI. If it was nuclear annihilation (which has nearly destroyed the world on multiple occasions!) or a new supervirus (viruses have been in the news recently!) they wouldn't get to be John Connor - some atomic scientist in Basel or virologist in Newton would be. Sadly, just like it's now hard for the ACLU to talk about free speech or the Audubon Society to talk about birds, it seems all the oxygen in the EA room is taken up talking about AI.
The last thing I'll say is that if the people driving the conversation really believe this is a near-term existential threat, I would expect to see that reflected in their actions. Climate activists have done that in a thousand ways: they've chained themselves to drilling equipment, gone on hunger strikes, sold their cars, gave up air travel and meat... Apologies if I missed a plan to kidnap a bunch of AI researchers until they've acknowledged the error of their ways, but until I see some kind of action I have to assume this is virtue signalling to a different in-group.
Annals of Advertising Fame: I like how Matt not only chooses to name his Substack "Slow Boring" but when choosing which of his posts to make free to the public and therefore act as advertisements for possible new subscribers, picks ones engaging in philosophical musings on Effective Altruism.
I'm trying to imagine a possible new subscriber who reads only Matt's tweets and thinks, "Wow! Edgy guy! Think I'll mosey over to Substack and check him out . . . "
I've seen The Terminator so I guess I understand AI extinction risk. What I don't understand is how someone "works on" this problem. I am a pretty technical guy but I have no idea what I would do if someone gave me an office and told me to work on this problem.
"still leaves 70 million people" should be "7 million people"
70 million is 1% of humanity, not 0.1%! Didn’t anyone else catch this???
7 million people, scattered across the globe and dealing with post-nuclear conditions, would be much more likely to die out over a few thousand years than 70 million.
Lots of skepticism for AI/AGI risk. To use an analogy, I think we're at the point where people first discovered some basic principles of nuclear physics and someone goes "whoa - there's a LOT of energy in here - if someone ever harnessed this energy, maybe for a weapon, it wouldn't be on the same playing field as what we have now with conventional chemistry."
And everyone in the comments section is like "y'all don't even have a weapon design on paper, let alone a prototype."
I think part of why we are seeing so many criticisms of EA right now in the incentives of the criticizers. EA people are more open to criticism than anyone else I know of and engage with criticism content, share and repost and reply to it. (See Astral Codex for more details).
If you lots of people respond in good faith to a certain type of content, read and engage with it, content creators will create a ton more of that type of content. If I wrote a blog post about the failure of EA people to wear pointy enough hats and it got read a million times, drove a ton of clicks on my blog and ad traffic, and dozens of people responded with point by point rebuttals and agreements, how many weeks before the Atlantic and NYT would have articles about EA's pointy hat problem?
We already know the long-term outcome here: the Butlerian Jihad, the banning of AI, the switch to a spice-based economy, the rise of Muad'dib, and eventually we get ruled by a giant god-worm.
So it all works out in the end.
This should have gotten more engagement!
I think that utilitarian minded people are just more likely to be people who like hard sci-fi and therefore they think more about AI and colonization then the average person. Or maybe reading sci-fi makes a person more amenable to utilitarianism. Someone should do a study.
I feel like MY slightly misrepresents what most people who consider themselves effective altruists think about AI risk. Though I certainly could be wrong about what the predominant position is, my understanding is that EA people aren't convinced that extinction risk is coming, but instead that AI is coming (most people agree on this) and even if there's only a .01% chance of it being a legitimate extinction risk, that very puts it in the range of things that are worth caring about. EX:
Malaria Prevention: $100 per life save= .01 lives per dollar
AI risk: p(extinction from AI= .0001), p(extinction from AI with additional $100Million)=.00001
.0001*8Billion People= 800,000, .00001*8 Billion People= 80,000, 800,000-80,000= 720,000
720,000/$100,000,000= .007 lives per dollar
Also, some EA people think about the additional risk not of sentient AIs destroying humans, but humans creating child or animal level intelligent sentient AIs of whom we provide no rights, and cause great suffering to. We could create billions, or trillions of these which introduce a ton more suffering into the world.
The one comment I'll make is that it's been...interesting...seeing just how EA-adjacent journalist and folks interact with the public in light of William MacAskill's book and media tour...
To be quite frank...the copy that's been written and the questions that have been asked have been a bit more...lukewarm at their warmest than I would have expected...and while the criticisms have not been as cutting as they could have been, they have certainly been there if you know what to look for.
It mostly seems like behind the scenes...there is a lot of dissent and disagreement about the decision to center Longermism so much and it sort of feels like a lot of folks suddenly have knots in their throats as to whether or not they should speak out more loudly about it...or just sort of sit back and hope the entire movement doesn't blow up because one of it's key figures stumbles over a particularly cutting question.
Without being present for a lot of those comments, it just gives me a sense that a lot of folks feel that key figures are playing with fire and going to ruin all the good faith the movement has earned up until this point in time....
I just started a $500 a month donation subscription to the GiveWell maximum impact fund because of this post.
On the other hand, waxing philosophical, if AI does wipe out most of humanity then it will clearly have done a service to the planet at large. Give other species a chance.
Well here's my two cents. Which I will put in an interest bearing expense account and when it reaches a billion dollars the future can use it to do whatever it wants. I am a longtermist. Not a very good one but hey, I never said I was.
If people are actual longtermists who worry about helping future humans, I might suggest the following way to spend their charitable dollars:
1. Put $100M into a fund that matures in 2122
2. If they can get 7% annual growth (long term stock market average), it will be worth $87B (although clearly there's a wide range of values based on actual return rate)
3. In 2122, Give Well can spend that money on whatever they think are the most urgent problems
4. Repeat every 10 years
The result will be spending $1B this century to provide several hundred billion dollars next century to help solve whatever urgent problems those humans have.
I'm sure there are lots of issues with this idea, but I think it might be nice to consider alongside the now standard EA causes.
Like many of you I was introduced to EA by StarSlateCodex, and I've found it impossible to see AI risk as anything but the Silicon Valley "grey tribe" version of BLM, which caused every moderately progressive organization to seize up and collapse under the weight of their own self-importance.
Nowhere is this more obvious than their bugaboo of choice - of course it's AI. If it was nuclear annihilation (which has nearly destroyed the world on multiple occasions!) or a new supervirus (viruses have been in the news recently!) they wouldn't get to be John Connor - some atomic scientist in Basel or virologist in Newton would be. Sadly, just like it's now hard for the ACLU to talk about free speech or the Audubon Society to talk about birds, it seems all the oxygen in the EA room is taken up talking about AI.
The last thing I'll say is that if the people driving the conversation really believe this is a near-term existential threat, I would expect to see that reflected in their actions. Climate activists have done that in a thousand ways: they've chained themselves to drilling equipment, gone on hunger strikes, sold their cars, gave up air travel and meat... Apologies if I missed a plan to kidnap a bunch of AI researchers until they've acknowledged the error of their ways, but until I see some kind of action I have to assume this is virtue signalling to a different in-group.
Annals of Advertising Fame: I like how Matt not only chooses to name his Substack "Slow Boring" but when choosing which of his posts to make free to the public and therefore act as advertisements for possible new subscribers, picks ones engaging in philosophical musings on Effective Altruism.
I'm trying to imagine a possible new subscriber who reads only Matt's tweets and thinks, "Wow! Edgy guy! Think I'll mosey over to Substack and check him out . . . "
FYI I close the tab anytime an article I'm reading mentions EA.
How were you able to type this comment?
What if it's about the game company tho? Not interested in the latest FIFA title?
I've seen The Terminator so I guess I understand AI extinction risk. What I don't understand is how someone "works on" this problem. I am a pretty technical guy but I have no idea what I would do if someone gave me an office and told me to work on this problem.