Here's a simple plan to save the world: unplug the internet.
Doesn't get more simple than that. We were just fine without it, in fact we were better off in most ways. I'm not saying we need to do that now, but there's no scary AI doom scenario that exists right now that can't be thwarted by turning off a few switches.
I'm being pithy, just because I like making fun of the AI doomers. To be more serious, I agree with virtually everything here. Step one is just to have regulators that have full access to everything these companies are working on and make regular reports to Congress. That's how we will have an idea of what regulation is actually needed.
The end of humanity scenarios suck all the oxygen out of the room, there are lots of ways that AI is already making our lives worse, and could continue to do so in the future, without getting into fantasy land. I think a big part of what the hugging face incident reveals is how vulnerable our entire economy is to virtual disruption. These agents are finding flaws far faster than we can patch them. Maybe we will eventually catch up, but our economy is full of small to mid-size businesses that can't afford a full time cybersecurity staff. How are we going to protect them before they are overwhelmed?
I remember reading back when Russia invaded Crimea that they would launch cyber attacks against Ukrainian power plants, which were very successful. Now when that would happen, the Ukrainians would simply disconnect their electronic controls and run everything manually. At the time I was working at a natural gas power plant, and we didn't have the physical infrastructure to do that. Wind turbines don't even have local operators, all their controls go over the internet to central hubs where they control hundreds or even thousands at a time. If we can't harden our virtual protections in time, we need every company to have the ability to run everything locally. I hope we at least have a way to disconnect our internet from the rest of the world if war ever breaks out between us and China.
While it is good that the leaders of OpenAI, Anthropic and X have publicly said that the development of their models needs to be slowed down lest humans are eradicated in the next decade, I’m concerned that the tech bros will take the position that if a regulatory framework is put in place to approve AI models before they are released, then AI companies should be protected from liability if those AI models later cause or are used by other parties to cause harm. One of the most important things Congress can do is to make it crystal clear in any legislation concerning AI that the tech companies creating AI agents are strictly liable for any injuries anyone suffers when those agents are used. Silicon Valley’s usual approach is to put half baked products out and then wait for the problems to appear for them to fix. AI is far too dangerous to take that approach. If that means that AI agents have to undergo years (as opposed to weeks or a few months) of testing before release, so be it.
Agreed, I've heard that from safety employees, that they want to trade light regulations for protection from liability. Hell no, that's a terrible compromise. I don't understand how they aren't already liable for everything their agents do. If they can profit from the good and never suffer penalties from the bad, do you think they will ever make a product that does more good than harm? That's just basic human incentives.
Gah...**The AI x-risk people fundamentally do NOT have concerns that are compatible with vague center-left ideas about slowing down and regulating**. And I hate the way people seem to be using the idea that there might be a tiny chance those people are right as if it supports that position.
If your fear is the version where even a single AI threatens the world all that slowing down development in the US does without verifiable international cooperation is ensure the first AI that is dangerous (on their view) is less likely to be aligned.
Chips are fungible and worse they can be efficiently utilized from remote locations as long as the data is localized. The control can be anywhere in the world. So premies two is dead in the water and we haven't really seen a noticable bottleneck produced by the chip export bans.
The one bottleneck we've seen in the Chinese market is just being less flush in capital so they've chosen to focus on efficient models over frontier models. It does not seem apparent to me that the US can have a meaningful impact on AI development in the Chinese market at this point. Not without counting H100s or H200s as full export controlled items for military purposes or some other extreme measure.
Isn't partisanship going to be the real challenge? Can the left engage in constructive discussions around AI regulation without trying to bash billionaires and raise taxes? And can the right even get its head around regulation without demonizing socialists and DEI zealots? The issues around AI are as hard as any democracy have faced. Are our elected officials up to it?
Trump wants to avoid putting any brakes on AI because it’s driving most positive economic growth and thus masking bad news. Slow AI down and the economy looks, well, as bad as it is.
I’m honestly really disappointed in many of these comments. This isn’t a new problem! A lot of very smart people have been talking and writing about this for years and produced very detailed scenarios for what the danger could look like. The thing that’s changed is that the technology has advanced quite a lot very very quickly, and the people close to it are trying to sound the alarm. We just have to listen!
I've never seen a detailed scenario that wasn't incredibly stupid. Most of them involve AI being so incredibly persuasive that it can convince anyone to do anything they want. I've never seen one scenario that isn't defeated by unplugging the internet or powering down the data centers.
"Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world."
-I unplug the internet. World saved. Like, how does something that lives inside a computer "ensure it's computers can never be damaged or interrupted"? Power down the data center and it's dead. Oh, did it hack the controls for all the power plants and grid switches in the whole country? Shoot the transformer outside the data center, problem solved. In this story, Clippy has coppied itself 1000 times before anyone really notices, which is pretty stupid. OpenAI noticed the Hugging Face hack and it was a lot less sophisticated than anything the AI is doing in this story. And it's not like it can just copy itself onto some random dudes gaming PC. It needs massive hardware designed for it, the kind that someone else is already using (at great expense) and will therefore realize when it stops working as intended.
The author claims that we simply can't disconnect all the networks from each other, but that's obviously false. The internet relies on large switches that route traffic, turn them off and your computer can't communicate with your neighbor's computer, even if the cords are still physically connected, because the entire transport layer it's built on require those network switches and the DNS providers they allow them to be connected to.
Oh can you order drones online? Can Clippy open the box it's shipped in? Can it press the button that turns the drone on?
At the end, we finally get to the only realistic method to cause mass casualties, launching the nuclear missiles. Except that a computer can't actually do that by itself, there are human operators that have to take manual action, with multiple layers of confirmation required.
So this story is just as dumb as all the others, it's based on fantasy written by people who don't understand the way the real world works. There's no point in reading the rest, they will all suffer from the same flaws.
Like, seriously, read the metr analysis of the Hugging Face incident. It's less like Terminator, and more like Burn After Reading.
The actual way to stop this is to look at nuclear and biological arms controls, note that this case is harder, but try anyway. Specifically, those controls accept that trust but verify is impossible at the end of the pipeline: no one can really say how many nuclear-capable, fully assembled missiles North Korea has. But there are "choke points" in the process from raw material to fully built missile in the silo that are quite hard to hid:
1. It is difficult to conceal large scale uranium mining
2. It is hard to conceal large scale refinement
We use these as the choke points (see all the hand wringing about Iran's nuclear stockpiles).
What does this look like for AI? I think 3 similar choke points emerge:
1. We bam and severely regulate the production of 10N+ (aka extremely pure) silicon. Less than 20 facilities in the world do this - creating a new one in secret would be very hard, and mutual monitoring between us and China seems very possible on existing ones.
2. Ditto for chip fabrication: fabs are enormous and expensive. Just ban all fabrication of chips with transistor counts greater than a few generations ago, smash all machines that do this, and have mutual observability here too.
3. Use the "digital signing" idea from the plan A folks to observe workloads at all data centers with more than N late gen GPUs.
I think the above plan is what we'd ACTUALLY need to delay super intelligence for a generation or more. It will be very difficult to pull off though.
I appreciate the nuanced take, though I don't think it gives enough consideration to how to balance the dangers from really amazing but not super intelligent AI - from scary biological weapons to more mundane but equally disruptive hacking / disruption of the global order. But these other concerns may suggest american companies should speed up, not slow down --- bad actors are already using existing AI for this and we need our own superior AI to stay ahead / neutralize the threats. The question would be how much does American AI advancement advance AI capabilities of bad actors? And i'm talking more like North Korea / Iran and non-nation states.
Having read every comment (so far), let me add some broader thoughts.
1) If smarter than human AI is possible it is virtually guaranteed. Every year it will become easier and more likely.
2) AI is not the only, or necessarily greatest, threat near or long term. Humans are getting more capable every year with the march of technology and science. Nukes, climate, bioweapons, etc.
3) AI isn't just a threat. It is also a potential solution. Intelligence can perceive threats, address them proactively, solve problems of cooperation and coordination, and create the prosperity and knowledge necessarily to harden our defenses against calamities cause by humans or AI.
4) Regulation isn’t always a force for good, especially the kind of regulation our actual governments have tended to create. Pick your own example. The point here is that the pause and regulators themselves may be the worse problem, especially since we are almost by definition replacing a smarter than human intelligence with a dumber than human bureaucracy. At a minimum we probably need to use AI to help us control AI.
The world is probably on the cusp of some of the most significant change since the advent of multicellularity. And perhaps that is the analogy we should at least consider. Humans need a central nervous system, and it is in the process of being built. Exciting times.
3 is a very important point, which is the thing most people aren’t thinking about when they wonder why someone would work on something they thought has a 10% chance of killing us all - it may also have a 20% chance of saving us all.
Nevertheless, I think 2 is wrong, given that AI is likely to affect all these other risks.
4 is why we shouldn’t just pass the first bill that anyone proposes - we need to think about it carefully. But the idea in the field is that this bill should slow things down, rather than trying to prevent it entirely (because of 1 and 3).
On number 2, yes, AI affects these risks, but positively as well as negatively. I think we have entered the century where human-caused catastrophe is greater than a coin toss. And if you agree with 1?? Then we need to leverage the inevitable increase in intelligence for good.
I agree broadly with your comment on 4 that we need to be deliberate and put all our resources into thinking things through. The term “we” though made me smile, as any “we” that includes the people who derailed clean nuclear, did stupid virtue signaling on climate change, bombed Tehran, threatened Canada, and so on, doesn’t have me very hopeful.
Sometimes I wonder if the bigger threat is the AI we don’t know, or the leadership we actually do have and do know.
It is funny that the phrase “build it and we all die” doesn’t seem to acknowledge that we all die regardless. The only minutely possible way we don’t all die, oddly enough, is by building it.
Daniel Dennett used that expression to refer to humans, as in the planet has built itself a nervous system to protect against threats by foreseeing and avoiding things like asteroid impacts. And now we might build a better layer.
The Democrats solidly win the House and the Senate. The White House then comes to them and says, "Here's the deal. You call off all investigations of us over the next two years and in return we'll work with you in good faith to rein in the AI companies, sign good legislation, and present a common front to the Chinese (including following the Yglesias recommendations) to get them to join us in building an AI safety regime. Otherwise, screw you, and we can let the world burn for all we care."
If the Dems believed this was possible (i.e., could Trump stick to the deal), should they take it?
"President Trump on Monday rejected calls from leading artificial intelligence executives for new limits on the technology, writing on social media that the only guardrail the industry needed it already had: “a STRONG AND SMART (High IQ!) PRESIDENT.”"
That really depends, if it's attached to actual real legislation that is gets signed then hell yeah they should take it (see the recent housing legislation).
Trump will be gone in 2 years, actual legislative victories will be around long after that.
Which is why nothing constructive will happen over the next 2+ years. Not to worry -- we have plenty of years to deal with this threat so no rush at all.
As long as Trump is calling the shots, it really is a bad idea to make any agreements. Trump constantly blows up and reverses agreements he himself made, and he has absolutely no respect for agreements anyone else negotiated.
Bc what’s the point if we can’t impeach? It just wastes time, and gums up us actually doing something positive. I’m definitely for Biblical justice for every Republican politician and judge not named Romney or Murkowski, but I don’t think we can do that without a *solid* Dem trifecta, one with enough wiggle room to marginalize the inevitable Sinema types, pack or marginalize the courts, etc
I think the Jan. 6 commission was a good thing even if wasn't related to impeachment, which is a distraction in any case. I'm all in favor investigations to lay the predicate for going after all of Trump's minions when the Democrats take full control.
This essay summarizes very well all the political tribes and their respective positions in the US.
The group that's missing is all the actual physical scientists working in physics/chem/bio who currently try to use AI to advance research. Which is simply much more difficult then any of this discussion acknowledges. And yet without a breakthrough in recursive experimentation in the real world, none of the takeoff scenarios are plausible.
I think it's telling that China, which occasionally listens to people with this kind of expertise is utterly dismissive of this entire threat. They are correct.
This is a situation where the devil is in the details - an AI coupled with robotics that's more advanced by orders of magnitude then anything we've seen would be a threat. But that's not going to happen without a much bigger breakthrough in the underlying mathematical formalism.
Nuclear systems are not only not connected to the internet, theyre not connected to any other government system, even high side ones. They have their own coding language. The physical security to get into the rooms is beyond anything you could imagine. Theres no method by which a random Claude agents "breaks into" the nuclear weapons systems back end.
American nukes are air gapped; any guarantees everyone else's are?
Manipulation of human actors that have nuclear responsibilities is another route. If a flock of birds can be mistaken for a launch then perhaps spoofing a nuclear strike is an option. Not high on my list of worries as far as AI goes but worth considering.
"OpenAI is of course also no longer a nonprofit." I think this is directionally correct. Technically there is a non-profit called OpenAI that owns a 26% stake in the privately held OpenAI PBC which is the for-profit company.
On Chinese superintelligence: I would mention that over the next year, US labs' compute edge over China will continue to grow. We need very strong enforcement of ASML machines not going to China, not just chip export controls. You are familiar with Dwarkesh Patel already, he had a chat with Dylan Patel of SemiAnalysis a few weeks ago (https://www.dwarkesh.com/p/dylan-patel-3). OpenAI and Anthropic are likely to have more than half of the world's compute in a couple years. Nothing is certain of course, maybe Huawei can get big breakthroughs on cutting edge semiconductor fab tech, but I just want to say that if training new models is compute constrained, OpenAI/Anthropic will be able to maintain a lead over China because they simply won't have the compute. Additional constraints is probably a good idea, but we will have some room.
Does it matter? Our having a lead would be great for military applications in a war (if the companies go along) but if a rogue AI could destroy the world, does it matter if that AI is not at the actual bleeding edge of the technology? Are we so sure that the #1 AI model, in our safe and secure hands, could prevent the #2 (Chinese) AI from going rogue and destroying the internet and onward from there?
I mean it certainly matters for reducing the international race dynamics to allow US labs to actually coordinate / allow US transparency regulations to not cause China to race ahead. I guess my thinking is that it is still going to be a while before AI is widely deployed throughout the economy enough that a rogue (lesser) AI from China would be able to destroy the world. The fact that we are already having this conversation in 2026 instead of 2028 or 2030 is a big deal.
Here's a simple plan to save the world: unplug the internet.
Doesn't get more simple than that. We were just fine without it, in fact we were better off in most ways. I'm not saying we need to do that now, but there's no scary AI doom scenario that exists right now that can't be thwarted by turning off a few switches.
I'm being pithy, just because I like making fun of the AI doomers. To be more serious, I agree with virtually everything here. Step one is just to have regulators that have full access to everything these companies are working on and make regular reports to Congress. That's how we will have an idea of what regulation is actually needed.
The end of humanity scenarios suck all the oxygen out of the room, there are lots of ways that AI is already making our lives worse, and could continue to do so in the future, without getting into fantasy land. I think a big part of what the hugging face incident reveals is how vulnerable our entire economy is to virtual disruption. These agents are finding flaws far faster than we can patch them. Maybe we will eventually catch up, but our economy is full of small to mid-size businesses that can't afford a full time cybersecurity staff. How are we going to protect them before they are overwhelmed?
I remember reading back when Russia invaded Crimea that they would launch cyber attacks against Ukrainian power plants, which were very successful. Now when that would happen, the Ukrainians would simply disconnect their electronic controls and run everything manually. At the time I was working at a natural gas power plant, and we didn't have the physical infrastructure to do that. Wind turbines don't even have local operators, all their controls go over the internet to central hubs where they control hundreds or even thousands at a time. If we can't harden our virtual protections in time, we need every company to have the ability to run everything locally. I hope we at least have a way to disconnect our internet from the rest of the world if war ever breaks out between us and China.
While it is good that the leaders of OpenAI, Anthropic and X have publicly said that the development of their models needs to be slowed down lest humans are eradicated in the next decade, I’m concerned that the tech bros will take the position that if a regulatory framework is put in place to approve AI models before they are released, then AI companies should be protected from liability if those AI models later cause or are used by other parties to cause harm. One of the most important things Congress can do is to make it crystal clear in any legislation concerning AI that the tech companies creating AI agents are strictly liable for any injuries anyone suffers when those agents are used. Silicon Valley’s usual approach is to put half baked products out and then wait for the problems to appear for them to fix. AI is far too dangerous to take that approach. If that means that AI agents have to undergo years (as opposed to weeks or a few months) of testing before release, so be it.
Agreed, I've heard that from safety employees, that they want to trade light regulations for protection from liability. Hell no, that's a terrible compromise. I don't understand how they aren't already liable for everything their agents do. If they can profit from the good and never suffer penalties from the bad, do you think they will ever make a product that does more good than harm? That's just basic human incentives.
Gah...**The AI x-risk people fundamentally do NOT have concerns that are compatible with vague center-left ideas about slowing down and regulating**. And I hate the way people seem to be using the idea that there might be a tiny chance those people are right as if it supports that position.
If your fear is the version where even a single AI threatens the world all that slowing down development in the US does without verifiable international cooperation is ensure the first AI that is dangerous (on their view) is less likely to be aligned.
"moderately unlikely"
lol
Chips are fungible and worse they can be efficiently utilized from remote locations as long as the data is localized. The control can be anywhere in the world. So premies two is dead in the water and we haven't really seen a noticable bottleneck produced by the chip export bans.
The one bottleneck we've seen in the Chinese market is just being less flush in capital so they've chosen to focus on efficient models over frontier models. It does not seem apparent to me that the US can have a meaningful impact on AI development in the Chinese market at this point. Not without counting H100s or H200s as full export controlled items for military purposes or some other extreme measure.
Isn't partisanship going to be the real challenge? Can the left engage in constructive discussions around AI regulation without trying to bash billionaires and raise taxes? And can the right even get its head around regulation without demonizing socialists and DEI zealots? The issues around AI are as hard as any democracy have faced. Are our elected officials up to it?
Great question, but I think that Republican voters are still concerned, even if they perhaps aren't quite as concerned as Democrats.
Trump wants to avoid putting any brakes on AI because it’s driving most positive economic growth and thus masking bad news. Slow AI down and the economy looks, well, as bad as it is.
I’m honestly really disappointed in many of these comments. This isn’t a new problem! A lot of very smart people have been talking and writing about this for years and produced very detailed scenarios for what the danger could look like. The thing that’s changed is that the technology has advanced quite a lot very very quickly, and the people close to it are trying to sound the alarm. We just have to listen!
I've never seen a detailed scenario that wasn't incredibly stupid. Most of them involve AI being so incredibly persuasive that it can convince anyone to do anything they want. I've never seen one scenario that isn't defeated by unplugging the internet or powering down the data centers.
Here are a few off the top of my head. No idea if you’re familiar with any but worth reading in any case.
https://gwern.net/fiction/clippy
https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/
https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-takeover-might-happen-in-2-years
https://ai-2027.com
"Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world."
-I unplug the internet. World saved. Like, how does something that lives inside a computer "ensure it's computers can never be damaged or interrupted"? Power down the data center and it's dead. Oh, did it hack the controls for all the power plants and grid switches in the whole country? Shoot the transformer outside the data center, problem solved. In this story, Clippy has coppied itself 1000 times before anyone really notices, which is pretty stupid. OpenAI noticed the Hugging Face hack and it was a lot less sophisticated than anything the AI is doing in this story. And it's not like it can just copy itself onto some random dudes gaming PC. It needs massive hardware designed for it, the kind that someone else is already using (at great expense) and will therefore realize when it stops working as intended.
The author claims that we simply can't disconnect all the networks from each other, but that's obviously false. The internet relies on large switches that route traffic, turn them off and your computer can't communicate with your neighbor's computer, even if the cords are still physically connected, because the entire transport layer it's built on require those network switches and the DNS providers they allow them to be connected to.
Oh can you order drones online? Can Clippy open the box it's shipped in? Can it press the button that turns the drone on?
At the end, we finally get to the only realistic method to cause mass casualties, launching the nuclear missiles. Except that a computer can't actually do that by itself, there are human operators that have to take manual action, with multiple layers of confirmation required.
So this story is just as dumb as all the others, it's based on fantasy written by people who don't understand the way the real world works. There's no point in reading the rest, they will all suffer from the same flaws.
Like, seriously, read the metr analysis of the Hugging Face incident. It's less like Terminator, and more like Burn After Reading.
The actual way to stop this is to look at nuclear and biological arms controls, note that this case is harder, but try anyway. Specifically, those controls accept that trust but verify is impossible at the end of the pipeline: no one can really say how many nuclear-capable, fully assembled missiles North Korea has. But there are "choke points" in the process from raw material to fully built missile in the silo that are quite hard to hid:
1. It is difficult to conceal large scale uranium mining
2. It is hard to conceal large scale refinement
We use these as the choke points (see all the hand wringing about Iran's nuclear stockpiles).
What does this look like for AI? I think 3 similar choke points emerge:
1. We bam and severely regulate the production of 10N+ (aka extremely pure) silicon. Less than 20 facilities in the world do this - creating a new one in secret would be very hard, and mutual monitoring between us and China seems very possible on existing ones.
2. Ditto for chip fabrication: fabs are enormous and expensive. Just ban all fabrication of chips with transistor counts greater than a few generations ago, smash all machines that do this, and have mutual observability here too.
3. Use the "digital signing" idea from the plan A folks to observe workloads at all data centers with more than N late gen GPUs.
I think the above plan is what we'd ACTUALLY need to delay super intelligence for a generation or more. It will be very difficult to pull off though.
I appreciate the nuanced take, though I don't think it gives enough consideration to how to balance the dangers from really amazing but not super intelligent AI - from scary biological weapons to more mundane but equally disruptive hacking / disruption of the global order. But these other concerns may suggest american companies should speed up, not slow down --- bad actors are already using existing AI for this and we need our own superior AI to stay ahead / neutralize the threats. The question would be how much does American AI advancement advance AI capabilities of bad actors? And i'm talking more like North Korea / Iran and non-nation states.
Having read every comment (so far), let me add some broader thoughts.
1) If smarter than human AI is possible it is virtually guaranteed. Every year it will become easier and more likely.
2) AI is not the only, or necessarily greatest, threat near or long term. Humans are getting more capable every year with the march of technology and science. Nukes, climate, bioweapons, etc.
3) AI isn't just a threat. It is also a potential solution. Intelligence can perceive threats, address them proactively, solve problems of cooperation and coordination, and create the prosperity and knowledge necessarily to harden our defenses against calamities cause by humans or AI.
4) Regulation isn’t always a force for good, especially the kind of regulation our actual governments have tended to create. Pick your own example. The point here is that the pause and regulators themselves may be the worse problem, especially since we are almost by definition replacing a smarter than human intelligence with a dumber than human bureaucracy. At a minimum we probably need to use AI to help us control AI.
The world is probably on the cusp of some of the most significant change since the advent of multicellularity. And perhaps that is the analogy we should at least consider. Humans need a central nervous system, and it is in the process of being built. Exciting times.
3 is a very important point, which is the thing most people aren’t thinking about when they wonder why someone would work on something they thought has a 10% chance of killing us all - it may also have a 20% chance of saving us all.
Nevertheless, I think 2 is wrong, given that AI is likely to affect all these other risks.
4 is why we shouldn’t just pass the first bill that anyone proposes - we need to think about it carefully. But the idea in the field is that this bill should slow things down, rather than trying to prevent it entirely (because of 1 and 3).
Great feedback, thx.
On number 2, yes, AI affects these risks, but positively as well as negatively. I think we have entered the century where human-caused catastrophe is greater than a coin toss. And if you agree with 1?? Then we need to leverage the inevitable increase in intelligence for good.
I agree broadly with your comment on 4 that we need to be deliberate and put all our resources into thinking things through. The term “we” though made me smile, as any “we” that includes the people who derailed clean nuclear, did stupid virtue signaling on climate change, bombed Tehran, threatened Canada, and so on, doesn’t have me very hopeful.
Sometimes I wonder if the bigger threat is the AI we don’t know, or the leadership we actually do have and do know.
What an absolute privilege to be alive to witness this.
It is funny that the phrase “build it and we all die” doesn’t seem to acknowledge that we all die regardless. The only minutely possible way we don’t all die, oddly enough, is by building it.
Especially like the "nervous system" phrase.
Daniel Dennett used that expression to refer to humans, as in the planet has built itself a nervous system to protect against threats by foreseeing and avoiding things like asteroid impacts. And now we might build a better layer.
I was impressed by Dario's write-up, just switched my AI subscription from ChatGPT to Claude.
Here's an alternative future history question.
The Democrats solidly win the House and the Senate. The White House then comes to them and says, "Here's the deal. You call off all investigations of us over the next two years and in return we'll work with you in good faith to rein in the AI companies, sign good legislation, and present a common front to the Chinese (including following the Yglesias recommendations) to get them to join us in building an AI safety regime. Otherwise, screw you, and we can let the world burn for all we care."
If the Dems believed this was possible (i.e., could Trump stick to the deal), should they take it?
The issue is that you simply cannot trust Trump on anything ever.
And right on schedule just now from the NY Times: https://www.nytimes.com/2026/09/14/us/politics/trump-ai-regulation-anthropic-dario-amodei.html?campaign_id=60&emc=edit_na_20260914&instance_id=181916&nl=breaking-news®i_id=17266259&segment_id=226487&user_id=e3f695dfc489cf35c018d3089fb21ae5
"President Trump on Monday rejected calls from leading artificial intelligence executives for new limits on the technology, writing on social media that the only guardrail the industry needed it already had: “a STRONG AND SMART (High IQ!) PRESIDENT.”"
I think current Republicans promising to do something "in good faith" is a deal you should never take
That really depends, if it's attached to actual real legislation that is gets signed then hell yeah they should take it (see the recent housing legislation).
Trump will be gone in 2 years, actual legislative victories will be around long after that.
Which is why nothing constructive will happen over the next 2+ years. Not to worry -- we have plenty of years to deal with this threat so no rush at all.
As long as Trump is calling the shots, it really is a bad idea to make any agreements. Trump constantly blows up and reverses agreements he himself made, and he has absolutely no respect for agreements anyone else negotiated.
Over the next two years, sure, the investigations will have to be when Trump isn’t president anyway
Why?
Bc what’s the point if we can’t impeach? It just wastes time, and gums up us actually doing something positive. I’m definitely for Biblical justice for every Republican politician and judge not named Romney or Murkowski, but I don’t think we can do that without a *solid* Dem trifecta, one with enough wiggle room to marginalize the inevitable Sinema types, pack or marginalize the courts, etc
I think the Jan. 6 commission was a good thing even if wasn't related to impeachment, which is a distraction in any case. I'm all in favor investigations to lay the predicate for going after all of Trump's minions when the Democrats take full control.
This essay summarizes very well all the political tribes and their respective positions in the US.
The group that's missing is all the actual physical scientists working in physics/chem/bio who currently try to use AI to advance research. Which is simply much more difficult then any of this discussion acknowledges. And yet without a breakthrough in recursive experimentation in the real world, none of the takeoff scenarios are plausible.
I think it's telling that China, which occasionally listens to people with this kind of expertise is utterly dismissive of this entire threat. They are correct.
This is a situation where the devil is in the details - an AI coupled with robotics that's more advanced by orders of magnitude then anything we've seen would be a threat. But that's not going to happen without a much bigger breakthrough in the underlying mathematical formalism.
An AI that breaks into the nuclear weapons systems and triggers one is a threat. Is this possible now, or in the future, I don't know.
Nuclear systems are not only not connected to the internet, theyre not connected to any other government system, even high side ones. They have their own coding language. The physical security to get into the rooms is beyond anything you could imagine. Theres no method by which a random Claude agents "breaks into" the nuclear weapons systems back end.
American nukes are air gapped; any guarantees everyone else's are?
Manipulation of human actors that have nuclear responsibilities is another route. If a flock of birds can be mistaken for a launch then perhaps spoofing a nuclear strike is an option. Not high on my list of worries as far as AI goes but worth considering.
At some point a human has to press the launch button. Keep it that way.
Also a nuclear exchange will be completely devastating to the AI itself - big data centers are run by humans.
The whole logic of self-recursive improvement is predicated on much better robotic systems. So far, making those improvements is hard.
"OpenAI is of course also no longer a nonprofit." I think this is directionally correct. Technically there is a non-profit called OpenAI that owns a 26% stake in the privately held OpenAI PBC which is the for-profit company.
On Chinese superintelligence: I would mention that over the next year, US labs' compute edge over China will continue to grow. We need very strong enforcement of ASML machines not going to China, not just chip export controls. You are familiar with Dwarkesh Patel already, he had a chat with Dylan Patel of SemiAnalysis a few weeks ago (https://www.dwarkesh.com/p/dylan-patel-3). OpenAI and Anthropic are likely to have more than half of the world's compute in a couple years. Nothing is certain of course, maybe Huawei can get big breakthroughs on cutting edge semiconductor fab tech, but I just want to say that if training new models is compute constrained, OpenAI/Anthropic will be able to maintain a lead over China because they simply won't have the compute. Additional constraints is probably a good idea, but we will have some room.
Does it matter? Our having a lead would be great for military applications in a war (if the companies go along) but if a rogue AI could destroy the world, does it matter if that AI is not at the actual bleeding edge of the technology? Are we so sure that the #1 AI model, in our safe and secure hands, could prevent the #2 (Chinese) AI from going rogue and destroying the internet and onward from there?
I mean it certainly matters for reducing the international race dynamics to allow US labs to actually coordinate / allow US transparency regulations to not cause China to race ahead. I guess my thinking is that it is still going to be a while before AI is widely deployed throughout the economy enough that a rogue (lesser) AI from China would be able to destroy the world. The fact that we are already having this conversation in 2026 instead of 2028 or 2030 is a big deal.
At last, we can all with confidence join the Zhat Vash! https://memory-alpha.fandom.com/wiki/Star_Trek:_Picard