This week on “The Argument” podcast, Jerusalem and I talked about A.I. risk and the possibility of superintelligence. Paid subscribers can watch or listen to the episode ad-free and read the transcript at the end of this post. Free subscribers can watch the episode on YouTube here or listen wherever you get your podcasts.
Concerns about catastrophic risk from A.I. are often dismissed as “science fiction” for the very good reason that such scenarios do, in fact, feature heavily in the history of science fiction.
The history of ideas is interesting and important, though. The question of A.I. risk is one that has been considered repeatedly by a large number of people over the past 150 years, including by many who are not science-fiction authors.
Of course, the fact that things occur in works of fiction is not evidence that they will occur in real life. But thinking and talking about fiction can clarify what we’re actually disagreeing about.
In smart cases against A.I. safety worries, like this recent one from Claire Lehmann, a lot of the words are wasted on questions that are not at all the crux of the issue.
Lehmann and Steven Pinker (someone else I agree with on most issues but not this one) are simply skeptical that superintelligence is possible. But this is the difference between not believing that humanity will ever be able to build spaceships with warp drives and not believing that a warp-capable human civilization would construct a large-scale United Federation of Planets. It’s one thing to debate whether something is or is not technically possible; it’s another to consider what impact it would have if it did occur.
I have no idea whether Pinker or Dario Amodei is correct about the possibility of superintelligence — they both know a lot more about cognitive science than I do — but I do believe that thinking about the history of fictional treatments of existential risk helps clarify what it is we are arguing about.
Superintelligent A.I. is hard to control
The oldest treatment of this I’m familiar with is from Samuel Butler’s 1872 book “Erewhon,” a somewhat satirical description of a society that has, among other things, essentially banned machines. They did so when Erewhonians realized that machines could undergo the same Darwinian process as animals, becoming more and more advanced over time until eventually — due to the shorter generation cycle — they would overtake humanity and displace us. The Butlerian Jihad in Frank Herbert’s “Dune,” in which humanity gets rid of all computers, is an homage to “Erewhon.”
In her final novel, 1879’s “Impressions of Theophrastus Such,” George Eliot takes Butler’s Darwinian argument seriously.