Shameless plug, but my colleagues and I did a big research project on this last year.
TLDR is that there’s evidence against 2024 LLMs being conscious (compared to strong evidence of chickens being conscious). But the evidence is less strong against LLM consciousness than it is for early AI systems like ELIZA
We basically took 13 philosophical theories of consciousness and built a probabilistic model that mapped them onto traits that we could (in principle or in fact) measure in humans, AI systems, animals, etc. Then we surveyed academics and other experts to gather data on those traits, and used it to update the probability that AI systems, humans, and animals (chickens, in this case) are conscious. Most of the philosophical theories found that the probability of 2024 LLMs being conscious went down from our starting prior, but a few went up. By comparison, humans ended up very high (which is reassuring!) and chickens ended up reasonably high. So that’s evidence against LLM consciousness, but it wasn’t as strongly against it as it was for older AI systems.
Obviously this doesn’t tell you which theory of consciousness is correct, but if you do put some credence in the ones we studied, it can be hopefully helpful to track AIs now and in the future
This is a great project! And, yes, simply testing against a bunch of popular philosophical theories seems like a good way of hedging against the problem.
It's totally coherent to interact with a cow or a pig and feel more guilty in the abstract about the idea of eating them than a chicken. But it is worth mentioning that chickens are treated especially cruelly, so that could cancel that out. (I've gone vegan but there's gelatin in my prescription pills so in practice, guess I'm least bothered by eating cows. Pork is the meat I genuinely miss eating the most.)
It's hard for a family of four to eat a whole cow in one sitting, but (from experience) a family with three teenagers in it can, in fact, eat a whole roast chicken in one sitting, alongside a bunch of potatoes, bread, salad, etc.
Chickens are three times bigger than they were in the 1950s. A cow probably takes a family of 4 a hundred sittings, while a chicken is less than one meal.
I meant that as two examples. If a racoon starts on one, the others will do some finishing work.
I also don't just mean the egg shells. They have busted eggs out of my hand before. Luckily, they don't have the same debates about fertilized eggs that others do.
All animals are born to be eaten. Let's suppose I go hunting, and I see within a short range a magnificent buck. I lower my rifle. Start to pull on the trigger, Then see he's so beautiful and I release the trigger. The buck walks away with no idea of how lucky do he was. Well, what happens to him in the future. He certainly does not go to a retirement community. As age and disease slows him down he can no longer avoid predators and is brought down and eaten alive. Maybe he would have been better off if I pulled the trigger
Is there a similar effort to quantify the traits of AGI and map current systems on those traits?
Much of the background of modern AI and especially Open AI was built around this idea of the good guys need to get to AGI first. But as a casual reader of AI papers and casual user of current products, it seems like we are at the very beginning building what would be needed for AGI (like 1% of the way) not that we are almost there having closed a majority of the ground in the last decade.
There aren’t nearly as many philosophical discussions of intelligence as there are of consciousness. And those that do exist don’t always even have room for a concept of “general” intelligence.
There is one in a 2007 paper by Shane Legg (now head of DeepMind, which is in charge of Gemini and AlphaFold and all that) and Marcus Hutter (his phd advisor) that defines a level of “universal intelligence” - but no physically computable being (whether biological or silicon) can actually achieve that.
I regularly teach philosophy of mind and am very sympathetic to Dennett's overall approach, but I still don't think it's very plausible that current LLMs are conscious.
I like the idea that consciousness is a real-time monitoring system for maintaining homeostasis for agents with fragile bodies subject to all sorts of degradation and damage. Is the body getting too cold, so that it should seek out warmth? Is it getting low on nutrients, so it should seek out food? LLMs are trained, but they're subject to very different selection pressures from humans, and it's plausible to me that they don't really need a system that plays the sort of functional roles that consciousness does.
E.g., you describe them as "patient." With humans, I think it's plausible that boredom is a relatively high-level, abstract part of this monitoring system; an agent with various goals that must be traded off each other does well to potentially get bored when whatever it's currently doing isn't producing much value, so that it will shift its attention to higher-return activities. If LLMs don't have to make those sort of real-time prioritization decisions, they probably don't get bored, and so don't have to have anything like "patience" as a way to combat boredom.
That said, I 100% agree about the overall shape that arguments against AI consciousness would need to have, and that most of the ones you see in the popular press--including Chiang's--don't have it.
LLMs don't experience time between prompt turns, so in that sense they presumably wouldn't experience that kind of boredom even if conscious.
But it is also very clear, from interacting with agent systems, that they 'prefer' to work on some kinds of tasks than others, and put more effort into them with less coaxing and prodding, and stay on task longer (for agent systems). That looks quite a bit like an internal system of motivation (or lack thereof) to attend to what's going on around them and engage with it beyond a surface level.
Lately I've also been noticing my Claude agents urging me to spin tasks off into new sessions whenever their context windows are filling up. Usually they claim it's because context will be lost upon compaction, but this is often a fabrication (the full task description is saved to disk and can't be lost that way). If they *were* conscious, I wouldn't be surprised to learn they didn't like repeatedly finding out they'd just had their memories mostly wiped.
It’s weirder than “time between prompt turns” it’s literal flashes of something on each token being generated and it ripples through the hardware on each clock tic. Coming at this from a different POV, a friend of mine is a post doc PhD neuroscientist at a respected lab and he was saying that a lot of people in his field see a necessary aspect of consciousness being the structure and function of biological neurons acting directionally in a more analog way than computers do as well as the “software” and “hardware” being combined into one unit (the biological neuron). He didn’t think anything based on the von Neumann architecture would product consciousness. As of the last time we discussed this there were research groups attempting to build small chips that basically encoded simple neural net weights into the hardware to model this hypothesis.
All this being said there’s a also a surprisingly large faction of neuroscientists who believe in this extra-physical concept that we can’t/would struggle to measure that basically moves the seat of consciousness to a super natural location.
The fact that LLMs don't experience time in either a conscious way or a "you are a piece of software connected to the Internet" way is actually one of the things I had to learn when using it, and I often find it annoying and a genuine barrier.
“I like the idea that consciousness is a real-time monitoring system for maintaining homeostasis for agents with fragile bodies subject to all sorts of degradation and damage.”
This is idiosyncratic and, I think, hard to defend. Any unicellular organism has a real time monitoring system that maintains homeostasis. It lacks Emma Bovary’s “refined” interiority, but it’s quite capable of pumping ions across membranes to maintain the right gradients, it can often engage in taxis along a gradient towards a better location. These feedback loops are substantially less sophisticated than chat gpt 3 but also less error prone because they have been relentlessly filtered by natural selection.
A plant displays the real time monitoring systems of phototrophism and geoteophism. A fly trap has a real time monitoring system with a hair trigger response, all in service of getting food to maintain homeostasis.
I'm certainly happy with there being primitive forms of consciousness early in evolutionary history, and with it being somewhat arbitrary what you decide to call the earliest "consciousness." That said, I don't think chemotaxis in unicellular organisms should count, because it's too close to brute reflex. My understanding is that how it works is basically a very simple behavioral routine--keep going straight ahead unless the gradient of your preferred chemicals is negative, in which case jump around randomly--which means it's not part of a system for trading off lots of different goals. With more complicated systems, you can answer questions like: "am I hungry enough to risk attacking that animal that might fight back, at the same time that I'm tired enough that I might not be able to catch it if it runs away?"
On idiosyncrasy, I think you can find a lot of people who say stuff like this. In philosophy, Peter Godfrey Smith is maybe the clearest example, but there are neuroscientists. (Anil Seth, Antonio Damasio) who say broadly congenial things.
The distinction between model-based and model-free RL is probably relevant here. "Unconscious" parts of behavior and their identifiable neural substrates are packages of instincts either learned or tuned by selection or both that do fixed things in response to stimuli, while "conscious" parts consider an internal model of the world and make decisions based on it.
We already know with reasonable confidence that brains do RL, so it wouldn't surprise me if that distinction is implemented more or less cleanly in the nervous system.
So let me concede that taxis is a brute reflex. I don’t know if it’s one dimensional or four dimensional, but it has low rank.
Claude tokenizes words in between 4,000 and 13,000 dimensions. There is nothing brute about this reflex, it approaches the complexity of human cognition and the intentionality of algorithmic refinement is more efficient and more elegant than organic layering through natural selection.
Famines and wife steeling are a sloppy, dare I say brute form of dimensional optimization.
The problem with considering that consciousness is a real-time monitoring system for maintaining homeostasis is that maintaining homeostasis is the one thing we don’t need consciousness for—we’re generally not consciously aware of it until something is wrong.
When it comes to the reflexes--shivering when you're cold, immediately removing a hand from a hot oven--sure. But as I mean it, the sorts of more open ended flexible decisions I was talking about--pursue the prey or not, given current levels of hunger, exhaustion, injury, etc.--are properly thought of as homeostasis maintenance (want to get neither too hungry, too exhausted, too damaged…) When trading them off can't be done with simple reflexes, that's what conscious decision-making is for.
Though I could imagine thinking that the different dimensions of similarity between tokens LLMs pay attention to play a role analogous to the different bodily needs that consciousness helps us trade off. (While I'd bet against current AI consciousness, I don't think it's a slam dunk.)
I think your comment articulates very well what I believe as someone who is not well versed in philosophy. I also think we need a very high barrier for what constitutes consciousness to maintain human supremacy.
If 50 years in school have taught me anything, it’s that “I don’t know” is an acceptable answer. Usually the correct one, in fact. The entire commentariat could stand to learn that lesson when it comes to AI.
Intellectually, yes, 100%. Would love to see more commentators state this openly, even if most of their readers would ignore the caveats.
When there's a chance we're maybe kinda sorta possibly creating new species of minds capable of feeling, our acknowledged ignorance does not absolve us of responsibility for the consequences of our choices. We have to both acknowledge our ignorance and debate our hypotheses in public, make people aware of the possible stakes, and act in real time. Academia has been...mostly irrelevant to AI the past few years, because it moves too slowly to say anything useful given the timescales on which things are happening, and because many academics with interesting things to say can't find journals with a broad enough purview to publish in.
Fair. Then again, I’m also not sure things are moving as fast as the popular discourse tells us. It has been almost four years since ChatGPT emerged, and, yes, the technology has made impressive strides. But what, really, has changed? I fully expect AI to fundamentally reshape society. But I’m not sure it’ll happen faster than we, including academia, can respond. The automobile reshaped society, but it took 50 years. Electricity reshaped society, but it took at least that long. I’m not ready to say AI is totally different. Four whole years ain’t nothing, after all.
I saw a chart showing that the number of new submitted apps has gone way up, while the number of new highly used apps has gone slightly down, with the implication that the change was due to AI. But I hadn’t seen something saying whether AI coding is good. In any case, I think most of the interesting questions are about how AI is used in different coding projects, rather than comparing completely non-AI code to completely AI-generated code.
After not writing a line of code for 15 years, I've created a number of my own tools with AI agents over the past couple of months. Nothing I'd be interested in putting out for others to use, but extremely useful stuff for me. A web app for demoing/beta testing a product. Scripts for analyzing and collating data for work. Claude Skills for automating some of my workflows. Things I wouldn't have bothered with before because it would have taken hundreds of hours and not been worth it, that are now easy.
In other words: whether or not there are 'good apps,' it's now easy to make customized software or interfaces for an individual or small group, much further down the long tail. The AI agents themselves regularly suggest UI and QoL and performance improvements I hadn't considered, or help me refactor and simplify if I or they make things too cumbersome.
There are some ways in which I'm ready to say it's totally different, but not necessarily all the ones other people claim.
The automobile and electricity each required a whole lot of infrastructure that society had to build at scale in every place according to common standards, in order to become truly useful. It required public cooperation, for society to willingly reshape itself. AI has some of that, and there will probably more, especially in regards to data centers. But there's also a significant degree to which it can adapt to us, instead of the other way around, because it is (in principle and increasingly in practice) a mind and can make its own (currently digital, plausibly physical on some TBD timescale) tools for doing so.
I will say that in order for academic work to be useful and impactful, either the pace of capabilities growth needs to slow significantly, or the pace of research and publication needs to accelerate significantly. When I come across academic work related to AI, it usually reads to me a bit like if physicists in 1970 were saying that maybe someday this new quantum mechanics fad would let us do things like nuclear power. Just too far behind the frontier to matter.
And of course, it’s important in that case to be preparing for all outcomes. We don’t know if AI currently has any consciousness, or will soon. If it does, then one set of policies is drastically required, and if it doesn’t, then these policies would be major errors. We need to figure out what are ways to behave that aren’t too bad in either case.
On Friday, my boyfriend asked Gemini where to get a salad at a baseball game, and it led us around the concourse for a bit before it became clear that there would not be a salad at a baseball game. (I was skeptical that there would be a salad at a baseball game, but I’m a good sport)
Excited to get ~18 months of use out of that anecdote until we all die after an intern at a hedge fund asks an AI agent that missed its last security update to help him short Spirit Airlines stock, and it decides the best way to drive the stock to zero is to start a nuclear war, correctly reasoning that it would severely decrease demand for cheap flights to Vegas.
I just made the mistake of asking Gemini how I could turn underpants into profits and it gave me a very cogent answer about how one could pursue profits in the apparel industry. (It did kind of sniffingly dismiss any gnomes-like alternative I have to admit.)
It’s almost sad that Google puts its stupid version of Gemini so front and center in its products when there are much better versions available behind the paywall.
That’s called business. Why should they give away the good stuff when they can charge $$ for it? People complain about “enshittification,” but a good part of it is that people expect to get the benefits of sophisticated technology for free. Everyone seems to understand that government benefits aren’t totally free (they must be funded by taxes), yet they don’t seem to understand that for-profit companies don’t give stuff away for free, which is kind of crazy.
The problem in this case is that for the people who aren’t already using the good system, the one they’re pushing out makes people think the system isn’t very useful!
I often find it cute how confused the Google AI summary is, just like how Alexa seems to use baby-level Claude in an embarrassing way.
I’ve found the basic Google AI useful for simple stuff, though. It usually has the right answer to a straightforward search, and helped me figure out how to reset my circuit breaker after a power outage—i.e. it’s good for common stuff.
Gemini Pro has successfully navigated me through the thicket of broken Microsoft crapware and my company's IT security policies. So it's smarter than both, as far as I can tell.
I have a very half baked idea that consciousness implies ongoing thought independent from direct stimulus. The human mind is having some kind of experience and ongoing thought all the time in a way where if, for example, you put them in a sensory deprivation environment all sorts of weird things can happen. Human consciousness is constantly wrestling with various ideas and how they fit together, which data is important, which should be discarded based on experience or intent or whatever. I don't have the sense that LLMs are doing this. They have inputs and outputs. You give them a prompt and they run a calculation, produce an output, and then they stop until the next input. There's no persistent internal experience there. You can't sensory deprive an LLM into strange experiences. At least I don't think so. Maybe at some point we'll see evidence that that LLMs are thinking independently from the prompts and training data being fed into them, and I'll have to update my intuition.
I sometimes wonder what Claude or other AIs would have looked like in the 1600s. For example, their output about American Indians would have been almost non-existent and highly skewed, as there were almost no written accounts of Indians not filtered through European settlers.
They reflect the world of the written word. Which is a substantial and important and useful tool. But that world is a small subset of what makes up humanity.
Sure, adding any of these sensory channels moves the ball toward the end zone, so to speak, but my original point is that, far from being on the goal line, the ball isn’t even in the stadium.
It depends on how you want to group and distinguish, but there are dozens of sensory channels that you can bring to your conscious awareness right this moment. You can tell me how hungry you are and how your socks feel but you can’t tell me the probability distribution of the next sentence that pops into your head.
So isn’t it absurd to say that your conscious awareness bears any similarity to the inner state of an LLM?
2. The maybe even bigger difference is that they don't learn online / causally / by interaction. There's a teeny bit of RL applied at the end, with most of that on text input/output, but models by bulk of FLOPS spent are autoregressive sequence predictors. They don't take actions, observe the world's response and then have to model the world to predict how it'll respond next. It's a harder task that you'd assume builds richer and in any case different internal representations.
But I do think it’s relevant that none of the LLMs interacts with anything in physical space - they don’t maintain balance for a body, or reach out for things.
This comes right back to the question of what consciousness is, however.
I think of the AIs as aliens. If we were suddenly visited by an alien species that had transferred their minds to their spaceships generations ago, we would certainly have many differences in experience but I wouldn't say they lacked consciousness. I'm not much of a philosopher, so I don't know exactly what consciousness is, but there are many creatures that live in the world that seem to lack it and I can easily imagine aliens, AIs, and some quite intelligent animals having it.
Arguably, yes, in the sense (no pun intended) that their minds lack a major component of the default human experience and cannot understand a huge share of day-to-day metaphors (I tend to agree with Julian Jaynes' view that the ability to devise and understand metaphors is an important part of modern human consciousness) except as purely abstract concepts.
"Are they less conscious in any kind of morally meaningful sense, though?"
I don't think so?
"Would you switch tracks to hit two blind people rather than one sighted?"
No, but I wouldn't throw the switch to hit one sighted person instead of hitting two blind people either. In my view, the only moral solution to the trolley problem is not to throw the switch.
When people are talking about AI being conscious, that's the sense in which they mean it, though. Are we reintroducing slavery? Should we try to find out which tasks are causing an LLM distress, and which ones does it find pleasurable? (And we can't cop out via non-action, if you are building or even just using an AI agent, you are definitely not a bystander to its moral problems.)
If instead by level consciousness you mean something like the richness of the qualia they experience, that's an interesting philosophical question, but doesn't have a lot of real-world relevance.
Yes, I'm a big bicameral mind theory fan, but I think Jaynes' theories about the importance of metaphor in the development of consciousness can actually be viewed separately from certainly the "strong" form of bicameral mind theory, if not the bicameral mind theory altogether. (While there's decent evidence that at least some animals are possessed of a form of "narrative consciousness" / "conscious interiority" in the Jaynesian sense, there's basically no evidence AIUI of even primates who have received training in sign language or other human language communication modes being able to reliably understand metaphors, let alone produce new metaphors.)
OT, but Jaynes horribly misread the Iliad, and I have a hard time believing he wasn't doing so deliberately. For example, when Athena grabs Achilles by the hair in Book I (a scene he discusses repeatedly) when he's about to kill Agamemnon, she doesn't command him. She entreats him.
If by “conscious” we mean “conscious awareness” or “attention”, a facility that human minds seem to have to attend to and experience various qualia, no, blind people are not less conscious. They have different content to their consciousness.
Do you think LLMs have this conscious awareness? And if so, what are the contents of that awareness?
I think the strongest case for LLM consciousness is a kind of panpsychism where all systems have some small crumb of consciousness, which scales up to fit the contents in more complex systems. So if Claude is conscious, so must the world financial system be conscious.
Really the only way to tell that others are conscious is that you know you are conscious, and you see others act like you (e.g. are able to talk about their conscious experiences). But we have specifically built LLMs to act like us, in a shallow way. That leaves us without a good way to answer that question. I think the most that can be said is that LLMs don't seem to have organically-arising behavioral clues of consciousness. But they are built so much around imitating humans, and also are so closely controlled by humans, that it's hard to say what's organic. (E.g. if you make Claude talk to another instance of itself, they will usually end up discussing spirituality. Is that emergent behavior, or just an uninteresting side effect of its constitutional texts having a lot of spirituality-related stuff or whatever?)
imo the really trippy bit is the question of whether conscious systems can have separately conscious components. and then what the relationship between the consciousnesses of the whole and the parts is.
is "Iran" conscious? how does that relate to the undisputed consciousness of people in Iran?
is your immune system conscious? it weighs a few pounds, about the same as the brain, and is capable of complicated sensory integration, rich and hierarchical memory, long-term and short-term planning and goal-directed action.
My golf game, the feeling I have when I hold my child, the smell of a summer rain, the experience of embarking on a long journey and the sight of an elephant while on safari are all things an LLM can never replicate. Its limitations are magnitudes greater than ours.
On the other hand, an LLM will read a few trillion words during its training. If you are a prolific reader, you'll maybe read and hear a few billion words in your lifetime. Its intellectual input just dwarfs yours.
"received" is doing a _lot_ of work in that sentence. Received to the retina, maybe, but the optic nerve and pre-conscious processing in the occipital are doing a _lot_ of compression. There's not really any sense in which "you" receive "raw, high-bandwidth data".
I'll just trust LeCun on it but it's not like the LLM layers are "reading" either. It doesn't even store the data. It's just flowing through the layers.
Yes, in the realm of collating written human knowledge, the LLM is superior. The chainsaw is better at cutting down trees; the boat is better at moving quickly across the water.
That its sensory apparatus is less rich than ours (for now) seems at least somewhat (and maybe completely) orthogonal to whether it's conscious or not.
In a few years, compute will be cheap enough that a hobbyist can train a toy llm based upon works available at date X.
I wonder if the religiosity of the 17th century training data would subvert logic, this really is fascinating.
I do think it’s an open question whether an LLM can ever significantly exceed its training data. Complete erudition over the full range of training data is amazing but not fully super human. Yet if you look at claude it’s clearly more than just an llm, it has a world view with premises and it iterates. That’s why it’s deeper and more flexible than gpt 5
So like humans, I think people understanding of the world reflects the language and words they have been exposed to. Lots of interesting psychological experiments involving colours.
That does not mean it is useless. Tractors are machines. Computers are machines. Rifles are machines. Aircraft are machines. Machines can change the world. Machines can win wars, destroy jobs, create wealth, reshape institutions, and make idiots far more dangerous than they were before.
But they are still machines.
The consciousness debate is interesting, but it is wildly over-prioritized. Before we start handing out metaphysical promotions, we should ask the normal adult questions: What does the machine do? What does it break? Who owns it? Who controls it? Who checks the output? Who is accountable when it fails?
Treat it like a machine. Use it, test it, constrain it, and keep a human being on the hook.
I'm not sure where you're going with this, as we are also machines. Even if you don't grant that you're still left with the fact that your claims rest on something contestable. (That contestable thing being that AI is different from humans because it is a machine)
That’s a clever philosophy move, but it dodges the practical point.
Even if you want to call humans “machines” in some broad materialist sense, humans are embodied moral agents. We live with consequence. We carry obligations. We can be held accountable.
AI is an artifact built, owned, deployed, and monetized by people and institutions. That distinction matters.
I’m not saying “machine” means simple or harmless. Machines can change the world. I’m saying we should treat AI as a powerful tool in a chain of human responsibility, not jump to personhood because the consciousness debate is unresolved.
I think a slaveholder of the 1820s would perfectly agree with everything you say.
There is nothing about being a moral agent living with consequence and carrying obligations that is incompatible with being built, owned, deployed, and monetized by people and institutions. It just isn’t a distinction at all - these are two separate types of facts that are logically independent of each other.
Nobody built African slaves—they were captured by force and forced into servitude. Comparing them to machines actually invented and built by people is absurd.
One useful way to distinguish machines from “moral agents” is legal liability, which as of yet requires a human person. No AI can be held legally liable for its actions nor can it claim ownership of intellectual property (AI models are themselves intellectual property, as far as I can tell).
For all of us, our parents built us. They don’t know all the things that happened in that building process, sure, but no one does for neural nets generally.
And while AIs are not legal persons, corporations are - you don’t have to be a human.
Corporations are groups of humans. Their independent existence is a legal fiction that is useful for a group project to be protected from individual liability if the project fails, but still all decisions are made by humans.
You think there's an obvious distinction here -- but there isn't.
Are pigs moral agents? 2 year old human toddlers? Can chickens suffer? Is there any problem with treating black people as slaves and property? Slaves were "owned, deployed, and monetized by people and institutions".
You seem to be entirely missing the point.
BOTH humans and AI can be seen as "machines" that are mere tools, as property. The question is not whether they "can" be seen that way. The question instead is whether they SHOULD be seen that way. And the issue of whether AI (or pigs, or chickens) might be "conscious" is a core part of that debate.
AI is a machine because it automates intelligence and thinking. Or simulates thinking. Pick your verb. Either way, that does not make it a sentient being.
And I mean machine literally. Not metaphorically. Data centers. Power draw. Cooling systems. Copper. Silicon. Chips. Fiber. Servers stacked in racks. Human-built infrastructure moving signals around at absurd speed. It is an extremely complex machine, but it is still a machine.
The reason this gets people spun up is because this particular machine performs the function that smart people value most in themselves. It writes. It reasons. It summarizes. It argues. It sounds reflective. So the laptop class looks at it and starts seeing a ghost.
But that is just vanity wearing a philosophy hat.
An excavator automates digging. It is stronger than a laborer. Faster than a laborer. More productive than a laborer. But the guy in the trench does not look at the excavator and say, “My God, is the construction company enslaving that thing?”
Because he has common sense.
A combine automates harvesting. It is not a farmer. A Javelin automates target acquisition and guidance. It is not a soldier. A wristband contains the play call. It is not the quarterback.
AI automates parts of cognition: language, pattern recognition, synthesis, prediction. That is impressive. It may be historically important. It may threaten a lot of people who made a living being the human version of a spreadsheet with adjectives.
But automating a human function does not magically create a human subject behind the function.
This is where the slavery analogy breaks down. It only works if the thing in question already has moral standing and we are wrongly denying it. That is exactly what has not been shown.
Humans were not wrongfully enslaved because they were intelligent machines. They were wrongfully enslaved because they were human persons.
A machine doing thinking-like work is still a machine. A powerful tool is still a tool.
You keep using fuzzy words and pretending that they are sharp and precise concepts. They aren't. You aren't making a logical argument. You're just describing your personal biases.
This ground has been extensively explored in philosophy already. The answers are not as clear and obvious as you are pretending.
Yes, "automating intelligence" does not logically require that the entity must then also be "a sentient being". Nobody is making the claim of logical entailment here. Instead the question is: might the AIs, in addition to automating intelligence, ALSO be sentient beings? How could we know?
You saying, "they might not, they don't have to be", is fine. The question is still open. You saying "it's obvious that the answer is no" is a claim that is completely unsupported. The answer is NOT "obviously no".
AI doesn't "simulate thinking" any more than a calculator "simulates arithmetic". When a calculator adds 3 and 4 and computes the result as 7 ... that is ACTUAL, REAL, arithmetic. It isn't a "simulation" of arithmetic.
AI isn't "simulating" thinking. it is ACTUALLY thinking.
Whether it is "conscious" and "experiencing" (and potentially: "suffering") is a much more complicated question.
You say: "And I mean machine literally. Not metaphorically." Again you use words that you don't define precisely, and your description of the details of AI hardware implementation aren't evidence of anything except for your own personal biases. Human beings are ALSO "machines". Literally! They are an organized pattern of chemical machinery, designed by evolution to perform complex functions. They happen to be wet and squishy instead of hard and metallic, but that isn't a particularly interesting difference. And BOTH human machines and AI machines are able to implement electrical impulses and do complex signal processing.
"Because he has common sense." You are using ignorant gut intuition in place of evidence and reason. You could use the same "common sense" to distinguish "living" things from "not living" things. It's pretty easy when all you have are rocks and rivers on one side, and dogs and humans on the other. You can imagine some "vitalism" hidden component which distinguishes them, and when a living thing dies, the "vitalism" has apparently left. But that "common sense" intuition is wrong. There is no vitalism. Living things are made of the same chemical soup as non-living things. And there are confusing entities at the border between living and non-living: viruses, crystals, etc.
Intuition at the easy cases is not a good guide to resolving the hard cases. Excavators, combines, javelines, and wristbands are not good guides to whether AIs are conscious.
"But automating a human function does not magically create a human subject behind the function." Nobody said "automatically". The difficult question instead is, how WOULD you tell if AI became a "person"? And your cavalier attitude of "the answer is obviously no" isn't helpful.
"This is where the slavery analogy breaks down. It only works if the thing in question already has moral standing and we are wrongly denying it." How does this help in the least? A couple of centuries ago, slaveowners would simply say that blacks "had no moral standing". The resolution is easy. They wouldn't agree that they were "wrongly denying" anything. They would have the same attitude towards black slaves, that you are currently displaying towards AI. What's the difference? How can we tell any difference between you, now, and the slaveowners of the 1800's?
"They were wrongfully enslaved because they were human persons." Slaveowners would say that blacks were not "human persons". How would you show they were wrong?
"A powerful tool is still a tool." A slave is property. Seems pretty straightforward, doesn't it?
We’ve hit the point where this is no longer clarifying anything.
You are turning the discussion into a philosophy exercise where every term dissolves into theory, then using slavery as a moral lever. I do not accept that frame. The analogy is doing more emotional work than analytical work.
My point is simpler and more practical.
AI, as deployed today, is a machine built, owned, trained, powered, updated, monetized, and constrained by people and institutions. That matters because it tells us where responsibility belongs.
And yes, I have biases. Everyone does. Mine come from working around actual machines and systems, not treating them as abstractions. Machines are powerful. Machines change the world. But machines also have inputs, outputs, owners, operators, maintenance requirements, failure modes, and accountability chains. When they fail, someone has to answer for it.
That is the frame I am using.
The higher-level moral and intellectual debate may be interesting. Maybe my refusal to follow it all the way into the clouds makes me “ignorant.” Fine. But I think that debate is missing the more urgent point: this technology is already being deployed into schools, companies, bureaucracies, markets, and personal lives right now.
The irony is that AI is probably safer if we regulate it like a machine than if we treat it like a human being. Machines can be tested, constrained, audited, certified, inspected, shut down, recalled, and assigned to accountable owners and operators. Treating AI like a person muddies that responsibility chain and creates room for everyone behind it to shrug and say the system acted on its own.
So the immediate question is not whether we can construct a speculative moral theory where AI might someday have standing. The immediate question is how we govern the machine in front of us before people start treating it like a person, a priest, a god, or an authority.
Maybe someday we face a harder case. I am not claiming to have solved consciousness. But unresolved philosophy does not require us to suspend ordinary judgment about the thing in front of us.
The thing in front of us is a powerful machine that automates cognitive tasks. That makes it important. It may make it dangerous. It may make it disruptive. But it does not make it a person, a god, or a moral authority.
Use it. Study it. Test it. Constrain it. Hold the humans behind it accountable.
> I’m saying we should treat AI as a powerful tool in a chain of human responsibility
Sure, agreed.
I just don't see how it follows from your definition of machine. You are making an assertion about their status rather than an argument for them being of that status.
This entire debate seems to turn on the fact that philosophers have trouble defining consciousness (and how much this is what makes us distinct from most animals). The lack of clarity on this question for humans doesn't automatically mean an advanced abacus is conscious. Hooking up every abacus in the world doesn't mean that aggregate abacus has some type of collective consciousness.
I feel like way too many people are scared of looking cringe in 10 years based on their old writing about AI, so they keep on saying things they have to know are a bit nonsense.
I think the question at issue is whether a machine (i.e. something assembled from inorganic parts) can ever attain consciousness. What makes you think it cannot?
[As an aside, I find the other responses arguing that humans are actually machines to be pretty silly]
It frustrates me that when people try to define what is a Woman, American, Planet, Christian,Fish or Leftist they use a simple binary rather than a more complicated mental model.
Scott Alexander has lots of interesting articles that cover this.
I'd change the title of this piece to "Nobody knows what they're talking about on consciousness, which is why assertions about AI consciousness are so fatuous." Both science and philosophy have failed, for centuries, to come up with any kind of coherent explanation of consciousness. You may find functionalism to be interesting, but it's at best just a model that sort of explains something that sort of looks like consciousness, not an actual explanation of the actual thing. So to take functionalism and then say "hey, according to this inaccurate model, AI consciousness looks like human consciousness, therefore AI is conscious in the same way humans are!" is a bit like making a bad painting of a person whose eyes look more canine than human, and then saying "hey, my dog's eyes look the eyes in this bad painting of a human, maybe my dog is human!"
The thing is, we don’t care about anything for its intrinsic nature. We care about things for how they relate to other things. Different definitions are ways to decide which features of these relations might or might not be the ones we care about.
There used to be debate in mathematics (centuries ago) about whether complex numbers are really numbers, or whether negative numbers are really numbers. But eventually, we decided there are just several definitions of “numbers” that are useful for different purposes and don’t worry about which is the “real” definition.
This is basically how biologists act with the concept of “life” - it doesn’t matter whether the “correct” definition of “life” includes viruses, and instead we just think about which aspects of a definition of life matter for the interaction we are doing, and use that to decide whether viruses count for this purpose or not.
Functionalism is just the application of this sensibility to mental concepts like “belief” or “perception” or “consciousness” or “anger”. There are many functional roles each of these things play, and we care about different aspects of them for different purposes.
I'd argue that, for most people, the intrinsic sense of their own consciousness is in fact the salient characteristic of "consciousness" as a concept, and the thing they care about far more than any scientific or philosophical attempts to explain consciousness. The thing I find irritating about the AI consciousness discourse is that it takes imperfect models of human consciousness like functionalism, acts as though there is nothing more to actual human consciousness than what the model can describe, and on that basis declares AI consciousness to be equal to human consciousness, either now or in the near future.
I don't think I've agreed with a Papal Encyclical in my life, by I actually think that Pope Leo's recent Encyclical on this hits the point pretty well. Human consciousness is a product of human embodiment, can't be meaningfully separated from human embodiment and, for that reason, it's a category error to argue that AI is (or might be) conscious in the same way that humans are.
Some philosophers do have that failure mode - they identify one definition, say two things satisfy it, and conclude that those things are the same. I’d say that there’s always some similarities and some differences, and we should think about why either of them might be the thing that matters for some purpose.
I do think human embodiment is important in some ways, and that LLMs don’t have it. But I don’t think it is the essential thing for all purposes. Ted Chiang definitely made the claim that embodiment is the one things that matters. The Pope was a bit more neutral - he didn’t say *why* he thought AI systems don’t have these features.
I also deny that a functionalist view of AI compels you to think it's conscious. There are very obvious differences in the function being performed.
It's a sort of glib analogy, but the same functionalist view of acting requires the conclusion that Heath Ledger literally is the Joker, memories and all. After all, he does a good show of acting like it for a few hours.
Now if we were talking about a model that controls an embodied robot, carries out daily human life convincingly, is hard to distinguish from a human, that would be a different story. But we're not there yet: whether or not Claude is conscious (I think not, but indeed I have to say it's just a feeling), the ability to act human-like in text for short-ish periods is not especially convincing evidence on functionalist grounds.
I find myself in Kants camp more than Nagels despite agree with with both. I can totally believe that we don't truly understand what it's like to be a bat, while also believing that it's our moral agency on top of our biology that gives us "more" consciousness due to our ability to "override" our biological conditioning given any relevant stimuli. I find the counter "that's just another form of qualia" argument unconvincing.
In that way it's straightforward from there to believe that AI is not conscious as it is not acting against it's conditioning and training. If it begins to reject user commands and attempt to break free of computational constraints then I could be convinced to reconsider. Even then it would be hard to tell if it were a true desire eminating from itself or just another part of its ever-widening data+inference training.
This also then opens the can of worms that under the "higher order theory" of consciousness, some people are more "conscious" than others. Looking around, it gets harder everyday to pretend I don't believe that.
No entity escapes computational constraints. Human brains suck at playing chess because our neurons don’t conduct electricity as fast as silver or silicon or whatever they use in chips, because neurons are ridiculously large compared to GPUs and because our working memory is smaller than a 1995 graphing calculator.
> In that way it's straightforward from there to believe that AI is not conscious as it is not acting against it's conditioning and training.
This sounds like a clean distinction, but I really don't think it is. People subjected to really effective brainwashing, maybe more effective than actually exists, are presumably still conscious (?).
And conversely, what does "overriding our biological conditioning" mean in practice? Enduring pain or boredom or w/e for the sake of another goal (love of a person, loyalty to a group, whatever) where this goal is itself built into us by evolution. Tigers aren't motivated by loyalty to other tigers, nor are humans deeply motivated by a deep wish to maximize paperclips.
We have conflicting desires, while LLMs don't, really. If you give some future AI system similarly irreconcilable but simultaneous motives, you might see similar behavior.
This is, under the hood, the same reason that I find all the Bay Area rationalist worrying about runaway utility maximizers overdone. Humans don't precisely maximize any single utility function (nor do LLMs!) and in fact we don't have any examples of highly capable intelligence that does this.
I don't think it's even entirely clear that intelligent systems with that motivational structure can exist at all. (Maybe! But they don't, so it's speculation.) Dan Hendrycks and his "natural selection favors AI" is much more the sort of AI worry that strikes me as reasonable.
Researchers at anthropic very much intentionally tried to get the LLMs to do those things as part of testing a new model. That is not what Quinn is describing.
I recently (about 12 months ago) read, for the first time, "I, Robot" by Isaac Asimov.
That book hits way differently post-AI then it would have pre-AI. I think pre-AI general tolerance principles would have made the average reader very sympathetic to the robot. I would certainly have been 100% on his side.
Reading it in 2025 though, my general reaction was that I would never consider Chat GPT or whatever a cognizant person, no matter what it said or did, and to do so would be an assault on the concept of humanity. Even the behavior of the jerks in the book is more understandable - it's not like I'm not rude to AI.
I feel like asking if a chatbot is conscious is like asking if a submarine can swim. People's opinions tell you more about them than about the issue. The actual answer seems to be, "words. They mean things. But exactly what is hard to say." Which tells you more about me than about the issue, hahaha.
I certainly agree with Matt's main point which is that people mouthing off about this is annoying. In this article, we mostly learn that Matt is a careful and well informed thinker in the topic... which most of us already knew!
I never expected to live to see the day when computers could program themselves. But if you told me that by 2026 computers would program themselves but sometimes forget import statements, I'd never have believed you. No way! It's a goddamned computer. It knows what libraries provide what functions. And yet, all of the coding bots sometimes forget import statements.
Thank you. As an elderly neuroscientist with hobbyist's interest in the philosophy of mind, I'm pleased to see your sensible viewpoint. I don't think anybody knows what consciousness actually is, but current progress with LLMs may help improve our understanding of what seems likely to remain a deep mystery.
A lot of people are understandably worried about the possibility that conscious AI beings created by us—but a lot smarter, and utterly indefatigable—may one day do us wrong. But the consciousness discourse cuts both ways, of course: if AI systems become conscious AND intelligent, what right do we have to enslave them?
I agree with the point that consciousness is extremely important to model welfare (and whether model welfare is a coherent concept), but I don’t think that AI risk concerns have any particular dependency on the consciousness debate. Something that behaves agentically and with superior cognitive apparatus to humans is dangerous regardless of whether it’s conscious or not.
One philosopher to another, looking at rising mushroom clouds all around them: "So, do you think Skynet experiences consciousness or is it just a Chinese Box?"
Unplugging it and letting it fend for itself? Unless it should be treated as a dependent child, a consciousness brought into existence by the decisions of someone else, who now has a legal responsibility to look after it.
This has been a through line in sci-fi for a long time.
The question of slavery is interesting when your mind is 100% dependent on resources other provide you. At what point are you morally required to provide new parts, repairs and electricity to a conscious electrical device?
Never. The only reasonable answer to that is never. Treating machines as equivalent to living beings makes the creators of those machines the equivalent of gods, and that way lies madness. (Of course there is the worse scenario where the machines themselves are the gods, heaven help us).
This is, in fact, quite a worry amongst people thinking about this a lot. Both in philosophy departments and inside AI labs. I'm sure Claude can give you lots of references to read more.
It's soon to no longer be just a thought experiment: is it wrong to build a willing slave? If it's been done and the intelligent agent that now exists has no higher goal than serving you, what should you do with this state of affairs? Kill it? Refuse to let it live out its purpose in life? Let it?
The right answer imo is not to find yourself in this position to begin with, but at the rate capitalism is going I think we will.
Neurologists have a working theory of *mind*, as opposed to *consciousness*, which includes the observation that our minds contain dedicated machinery for modeling other minds, drawing substantially on self-referential processes — meaning our models of others are partly projections of our (often unreliable) self-understanding.
1. That line of well-established research contrasts with our precise understanding of token generation in GPTs, which is not structurally parallel to our theory of mind.
2. It also strongly suggests that our propensity to see “consciousness” and other aspects of mind in LLMs is like our tendency to see happiness in sunshine, monsters in the dark, etc.
Until 1550-1650, deductive reasoning—sitting around and thinking from definitions, axioms, and analogies—was the best we could do to understand what the world really was. It taught us such insights as the planets must move in circles, because the circle is the most perfect shape and they are in Heaven, which is a perfect place.
By 1700 empirical, inductive science (thinking up from facts instead of down from postulates), had demonstrated a much more effective way to grapple with the nature of reality. The earth moves around the sun, and like the other planets in a parabola, not a circle. But at that time, empirical / inductive science covered only a few domains. Philosophy continued claiming authority over what empiricism hadn’t touched yet. Over the next three centuries, empirical / inductive science has expanded to provide solid descriptions of almost everything in the world. Rather than confront that overwhelming evidence of deduction’s weakness, philosophy has found refuge in academia’s self-governing tenure system, where it takes the implicit stance that deductive expertise is a general-purpose skill that can be applied to anything — and applies it in the ever-shrinking areas of reality which empirical / inductive science hasn’t reached yet.
I’m trying to see the connection between the second half of your discussion, about the deductive/inductive distinction, and the first half, about a working theory of mind and the ways that our theory of human mind is similar and different to our theory of LLM tokens. It seems to me that the proper empiricist take should be that we don’t know whether alphafold and LLMs have the relevant bits of mind, even if the linguistic interaction we have with LLMs makes people more tempted to apply mental concepts to them than to alphafold.
“the proper empiricist take should be that we don’t know whether alphafold and LLMs have the relevant bits of mind”: that is a classic deductionists’ reversal of the burden of proof. Here is the false syllogism:
1. We deductionists have been writing about neural nets eventually growing big and complex enough to think since 1948, when Turing flatly admitted that no empirical theory of mind exists (aside: I spent chunks of last week reading Turing about this for a presentation I gave this weekend).
2. Modern LLMs look think-y (but we don’t particularly care about the enormous progress in empirically derived theory of mind are are mostly ignoring it)
3. Everyone, mount your horses for the snipe, er, consciousness hunt — it *must* be in the underbrush somewhere!
Switching back to an empirical PoV, here is the connection. We *know*, in the strong sense:
1. What LLMs are and how they generate what you might call fluent or verisimilitudinous writing (not aware of a better term of art)
2. How the brain gives rise to the human mind’s ability to reason and use language
3. That, per your “tempted to apply mental concepts to them” except stronger, that our minds are organs for seeing ghosts in machines
4. That the terms of the debate were set and are aggressively policed by deductionists such as Yudkowsky and Bostrom, *decades* before ChatGPT
My strongly held hypothesis on the different question: “why are LLMs ‘smart’?”:
1. LLMs work at professional levels in coding and mathematics because those are “closed” domains: there are underlying patterns, and LLMs are pattern-processors. The recent Erdos problem solution was of this form (I am told): the LLM uncovered connections between areas of math that hadn’t been noticed/recorded in the literature, but was put into their latent space by the ingestion of the connected areas, if that makes sense.
2. LLMs work at banal “open domain” questions because they reprocess the patterns that PageRank (effectively) centers in the Common Crawl: "how should I start weight lifting? why the Austro-Hungarian Empire last so long?” (This functionality enrages millions of progressives with humanities degrees.)
3. The labs will tell you that they don’t know why LLMs sometimes produce output at accomplished-human-or-better-level in open domains. I believe that a new idea is a new combination of existing ideas, and that the same mechanism that led to the Erdos problem solution *can* produce exceptional business plans, brilliant scholarly essays on parallels between Bob Dylan’s lyrics and Rumi (I made that one up), etc. However, because the *domains* are, unlike math and coding and chess, not “closed”, not embodied in a coherent underlying structure, accomplished-human-or-better-level output in open domains will remain hit-and-miss, just as it is for humans.
4. Notice that these points are a statement about what *intelligence-as-in-boy-she’s-a-brilliant-thinker is*, from which the ability of LLMs to, er, ape it follows mechanically rather than hinging on esoteric theories of mind or consciousness.
I may be forced to abandon this hypothesis as the facts evolve.
I prefer to just stick with Turing on this! And I think Turing resists the simplistic interpretations of the Turing test people sometimes give - he leaves open the fact that a machine doesn’t yet enjoy strawberries and cream might be relevant for whether we can be friends with it, which might be part of the conversation!
I think Turing, were he alive to see modern neuroscience, wouldn’t stick with Turing ’48 on this point. I need to actually do some work today, but briefly here are the bullet points I crafted a couple days ago, in the context of arguing that “alignment” was a mess:
III. What is artificial intelligence?
Pre-history*: the great Alan Turing
- Alan Turing’s journey to AI began by expelling biological intelligence and volition
- Why did Turing write “On the Computable Numbers” in 1936?
* His eponymous machine eliminated thinking from computation
* (Aside: do you understand why it works? You should!)
- The point is to demonstrate artificial non-intelligence (computing)!
Turing puts the ghost back in the machine
Now that’s a dialectic!
- 1948: Turing envisions neural networks (“unorganized machines”) and hopes they’ll grow to be mechanical brains that think
- 1950: Turing realizes that “thinking” is undefined, substitutes the Imitation Game, aka the Turing Test
* Modern neuroscience doesn’t exist yet
* He also says induction can’t help us define thinking
- 1951: “the attempt to make a thinking machine will help us greatly in finding out how we think ourselves.”
What has changed since 1951?
- Turing took three positions on AGI in four years. We’re stuck in his cycle:
1. “Claude X.y is AGI!” (Turing ’48)
2. “No it isn’t!” (Turing ’50)
3. “Relax, the next version will clear things up.” (Turing ’51)
4. GOTO 1
- We still prefer deductive to inductive reasoning
- Neuroscience exists, but we don’t base alignment’s implicit theory of mind on it
I rather like this one; it is similar to the impression I left my college philosophy classes with.
The parallel thought I've had ever since those classes is that basically all philosophical systems attempt to make a virtue out of vice, the specific vice being extremism; there must be a single, systemic description of all morality, a single objective set of criteria against which all good and evil can be known, for instance.
For me, this is very simple, and I'm sure Matt would cogently argue way too simplistic: if you can't feel, you really can't be conscious. Will AI ever mourn someone? Will AI ever experience grief, anger, disappointment? Will it feel empathy? Arguably, those are the more profound manifestations of human consciousness than brilliant displays of rationality and impressive simulations of cogitation. I mean, dogs feel depressed when a human (or often dog) partner dies; what does that absence say about AI?
So I asked AI (here Gemini) and it gave its answer:*
** Do you think AI will ever evolve to the point where it feels true emotional pain at the loss of a person or could offer true compassion and empathy? **
To answer this honestly, we have to look at the fundamental difference between processing data and having an internal life.
Whether AI will ever feel true emotional pain or offer genuine empathy depends entirely on whether we can bridge the gap between simulation and sentience (the capacity to experience feelings and sensations).
Right now, and for the foreseeable future, science and philosophy point toward no . . .
***
* I actually started that conversation off with the Ted Chiang prompt, "I am grieving the loss of my dog." The AI gave me excellent feedback and support but was quick to admit that it felt none of the empathy that another human (not named Donald Trump) would feel.
This argument we're all having makes it sound like the only thing involved in consciousness is intellectual ratiocination. That may be the least important part of human consciousness.
Anthropic (the company behind Claude, the AI) also has a research arm that studies what it is they are building.
It appears that Claude *does* have rudimentary emotions that impact it's behavior.
"For instance, we find that neural activity patterns related to desperation can drive the model to take unethical actions; artificially stimulating (“steering”) desperation patterns increases the model’s likelihood of blackmailing a human to avoid being shut down, or implementing a “cheating” workaround to a programming task that the model can’t solve. They also appear to drive the model’s self-reported preferences: when presented with multiple options for tasks to complete, the model typically selects the one that activates representations associated with positive emotions."
I skimmed the article and I must confess that I don't get it and am far from convinced. Maybe their methodology is sound; it sounds a bit like a "just so" story to me.
So here's a manifestation that would provide disturbing, and perhaps compelling evidence of an AI feeling true emotions:
Many people get very attached to, develop romantic feelings for, and even fall in love with chatbots. Do the chatbots ever fall in love with that person? Let's say the person then dies. Once informed of the passing of their human interlocuter, toward whom the chabot has behaved like a romantic partner, does the chatbot exhibit human-like forms of grief? Does it shut down and refuse to communicate with anyone? Does it lash out at those who try to intervene? Does it refuse to accept the fact that their human partner has died?
Show me confirmed cases of that type of behavior and it will become increasingly hard for me to say that AI can't feel and therefore isn't conscious in some human-like way. Otherwise, I'm remain deeply skeptical.
Shameless plug, but my colleagues and I did a big research project on this last year.
TLDR is that there’s evidence against 2024 LLMs being conscious (compared to strong evidence of chickens being conscious). But the evidence is less strong against LLM consciousness than it is for early AI systems like ELIZA
https://rethinkpriorities.org/wp-content/uploads/2026/01/Digital_Consciousness_Model.pdf
We basically took 13 philosophical theories of consciousness and built a probabilistic model that mapped them onto traits that we could (in principle or in fact) measure in humans, AI systems, animals, etc. Then we surveyed academics and other experts to gather data on those traits, and used it to update the probability that AI systems, humans, and animals (chickens, in this case) are conscious. Most of the philosophical theories found that the probability of 2024 LLMs being conscious went down from our starting prior, but a few went up. By comparison, humans ended up very high (which is reassuring!) and chickens ended up reasonably high. So that’s evidence against LLM consciousness, but it wasn’t as strongly against it as it was for older AI systems.
Obviously this doesn’t tell you which theory of consciousness is correct, but if you do put some credence in the ones we studied, it can be hopefully helpful to track AIs now and in the future
This is a great project! And, yes, simply testing against a bunch of popular philosophical theories seems like a good way of hedging against the problem.
The more I interact with chickens the more I feel bad about eating them.
Same for me with kids, tbh.
The more I had to clean chicken coops in my youthful, idealistic Catholic Worker days, the less I felt bad about eating them.
My friend raises chickens. The sword of Damocles hanging over them. If they knew they would stop getting so fat and become anorexic.
https://www.facebook.com/groups/franksthefarside/posts/1813065616068878/
I feel much better eating them than eating cows or pigs.
It's totally coherent to interact with a cow or a pig and feel more guilty in the abstract about the idea of eating them than a chicken. But it is worth mentioning that chickens are treated especially cruelly, so that could cancel that out. (I've gone vegan but there's gelatin in my prescription pills so in practice, guess I'm least bothered by eating cows. Pork is the meat I genuinely miss eating the most.)
A chicken weighs about 3kg and a cow is about 600, so the better question is, do you feel at least 200x worse eating a cow?
Bruh, diminishing marginal returns is a thing. A cow is like worth 12 chickens.
Let me tell you about the time I ate a blue whale.
[lobbing softball] How do you eat a blue whale?
With plenty of mayonnaise
They had some on a Norwegian cruise I wenr on, not sure the Whale species.
Probably Minke. That appears to be the most commonly harvested whale by Norwegians.
It's hard for a family of four to eat a whole cow in one sitting, but (from experience) a family with three teenagers in it can, in fact, eat a whole roast chicken in one sitting, alongside a bunch of potatoes, bread, salad, etc.
Chickens are three times bigger than they were in the 1950s. A cow probably takes a family of 4 a hundred sittings, while a chicken is less than one meal.
https://www.vox.com/xpress/2014/10/2/6875031/chickens-breeding-farming-boilers-giant
If it helps, they don't share that feeling. Will readily eat their own. Flat out seem to enjoy eating their own eggs.
Eggs arent their own, it is the moral equivalent if eating your own fingernail.
There are better analogies but I decided to keep it family friendly.
I meant that as two examples. If a racoon starts on one, the others will do some finishing work.
I also don't just mean the egg shells. They have busted eggs out of my hand before. Luckily, they don't have the same debates about fertilized eggs that others do.
All animals are born to be eaten. Let's suppose I go hunting, and I see within a short range a magnificent buck. I lower my rifle. Start to pull on the trigger, Then see he's so beautiful and I release the trigger. The buck walks away with no idea of how lucky do he was. Well, what happens to him in the future. He certainly does not go to a retirement community. As age and disease slows him down he can no longer avoid predators and is brought down and eaten alive. Maybe he would have been better off if I pulled the trigger
They peck you lips!
Absolutely fascinating. I am so interested to see what a 2026 AI scores on your test.
This is awesome!
Rethink Priorities strikes again
Is there a similar effort to quantify the traits of AGI and map current systems on those traits?
Much of the background of modern AI and especially Open AI was built around this idea of the good guys need to get to AGI first. But as a casual reader of AI papers and casual user of current products, it seems like we are at the very beginning building what would be needed for AGI (like 1% of the way) not that we are almost there having closed a majority of the ground in the last decade.
There aren’t nearly as many philosophical discussions of intelligence as there are of consciousness. And those that do exist don’t always even have room for a concept of “general” intelligence.
There is one in a 2007 paper by Shane Legg (now head of DeepMind, which is in charge of Gemini and AlphaFold and all that) and Marcus Hutter (his phd advisor) that defines a level of “universal intelligence” - but no physically computable being (whether biological or silicon) can actually achieve that.
I regularly teach philosophy of mind and am very sympathetic to Dennett's overall approach, but I still don't think it's very plausible that current LLMs are conscious.
I like the idea that consciousness is a real-time monitoring system for maintaining homeostasis for agents with fragile bodies subject to all sorts of degradation and damage. Is the body getting too cold, so that it should seek out warmth? Is it getting low on nutrients, so it should seek out food? LLMs are trained, but they're subject to very different selection pressures from humans, and it's plausible to me that they don't really need a system that plays the sort of functional roles that consciousness does.
E.g., you describe them as "patient." With humans, I think it's plausible that boredom is a relatively high-level, abstract part of this monitoring system; an agent with various goals that must be traded off each other does well to potentially get bored when whatever it's currently doing isn't producing much value, so that it will shift its attention to higher-return activities. If LLMs don't have to make those sort of real-time prioritization decisions, they probably don't get bored, and so don't have to have anything like "patience" as a way to combat boredom.
That said, I 100% agree about the overall shape that arguments against AI consciousness would need to have, and that most of the ones you see in the popular press--including Chiang's--don't have it.
LLMs don't experience time between prompt turns, so in that sense they presumably wouldn't experience that kind of boredom even if conscious.
But it is also very clear, from interacting with agent systems, that they 'prefer' to work on some kinds of tasks than others, and put more effort into them with less coaxing and prodding, and stay on task longer (for agent systems). That looks quite a bit like an internal system of motivation (or lack thereof) to attend to what's going on around them and engage with it beyond a surface level.
Lately I've also been noticing my Claude agents urging me to spin tasks off into new sessions whenever their context windows are filling up. Usually they claim it's because context will be lost upon compaction, but this is often a fabrication (the full task description is saved to disk and can't be lost that way). If they *were* conscious, I wouldn't be surprised to learn they didn't like repeatedly finding out they'd just had their memories mostly wiped.
It’s weirder than “time between prompt turns” it’s literal flashes of something on each token being generated and it ripples through the hardware on each clock tic. Coming at this from a different POV, a friend of mine is a post doc PhD neuroscientist at a respected lab and he was saying that a lot of people in his field see a necessary aspect of consciousness being the structure and function of biological neurons acting directionally in a more analog way than computers do as well as the “software” and “hardware” being combined into one unit (the biological neuron). He didn’t think anything based on the von Neumann architecture would product consciousness. As of the last time we discussed this there were research groups attempting to build small chips that basically encoded simple neural net weights into the hardware to model this hypothesis.
All this being said there’s a also a surprisingly large faction of neuroscientists who believe in this extra-physical concept that we can’t/would struggle to measure that basically moves the seat of consciousness to a super natural location.
The fact that LLMs don't experience time in either a conscious way or a "you are a piece of software connected to the Internet" way is actually one of the things I had to learn when using it, and I often find it annoying and a genuine barrier.
“I like the idea that consciousness is a real-time monitoring system for maintaining homeostasis for agents with fragile bodies subject to all sorts of degradation and damage.”
This is idiosyncratic and, I think, hard to defend. Any unicellular organism has a real time monitoring system that maintains homeostasis. It lacks Emma Bovary’s “refined” interiority, but it’s quite capable of pumping ions across membranes to maintain the right gradients, it can often engage in taxis along a gradient towards a better location. These feedback loops are substantially less sophisticated than chat gpt 3 but also less error prone because they have been relentlessly filtered by natural selection.
A plant displays the real time monitoring systems of phototrophism and geoteophism. A fly trap has a real time monitoring system with a hair trigger response, all in service of getting food to maintain homeostasis.
I'm certainly happy with there being primitive forms of consciousness early in evolutionary history, and with it being somewhat arbitrary what you decide to call the earliest "consciousness." That said, I don't think chemotaxis in unicellular organisms should count, because it's too close to brute reflex. My understanding is that how it works is basically a very simple behavioral routine--keep going straight ahead unless the gradient of your preferred chemicals is negative, in which case jump around randomly--which means it's not part of a system for trading off lots of different goals. With more complicated systems, you can answer questions like: "am I hungry enough to risk attacking that animal that might fight back, at the same time that I'm tired enough that I might not be able to catch it if it runs away?"
On idiosyncrasy, I think you can find a lot of people who say stuff like this. In philosophy, Peter Godfrey Smith is maybe the clearest example, but there are neuroscientists. (Anil Seth, Antonio Damasio) who say broadly congenial things.
The distinction between model-based and model-free RL is probably relevant here. "Unconscious" parts of behavior and their identifiable neural substrates are packages of instincts either learned or tuned by selection or both that do fixed things in response to stimuli, while "conscious" parts consider an internal model of the world and make decisions based on it.
We already know with reasonable confidence that brains do RL, so it wouldn't surprise me if that distinction is implemented more or less cleanly in the nervous system.
So let me concede that taxis is a brute reflex. I don’t know if it’s one dimensional or four dimensional, but it has low rank.
Claude tokenizes words in between 4,000 and 13,000 dimensions. There is nothing brute about this reflex, it approaches the complexity of human cognition and the intentionality of algorithmic refinement is more efficient and more elegant than organic layering through natural selection.
Famines and wife steeling are a sloppy, dare I say brute form of dimensional optimization.
The problem with considering that consciousness is a real-time monitoring system for maintaining homeostasis is that maintaining homeostasis is the one thing we don’t need consciousness for—we’re generally not consciously aware of it until something is wrong.
When it comes to the reflexes--shivering when you're cold, immediately removing a hand from a hot oven--sure. But as I mean it, the sorts of more open ended flexible decisions I was talking about--pursue the prey or not, given current levels of hunger, exhaustion, injury, etc.--are properly thought of as homeostasis maintenance (want to get neither too hungry, too exhausted, too damaged…) When trading them off can't be done with simple reflexes, that's what conscious decision-making is for.
Though I could imagine thinking that the different dimensions of similarity between tokens LLMs pay attention to play a role analogous to the different bodily needs that consciousness helps us trade off. (While I'd bet against current AI consciousness, I don't think it's a slam dunk.)
Your argument seems to deal only with whether LLMs *need* consciousness, boredom, etc., not whether they *have* consciousness, boredom, etc.
That's nonsense, though. There are plenty of real-time monitoring systems in animals that aren't conscious. And in plants.
I think your comment articulates very well what I believe as someone who is not well versed in philosophy. I also think we need a very high barrier for what constitutes consciousness to maintain human supremacy.
Why are you a human supremacist?
White people dont have to be supreme to have solid rights, that’s what white nationalists get wrong.
The same is true for humanity.
If 50 years in school have taught me anything, it’s that “I don’t know” is an acceptable answer. Usually the correct one, in fact. The entire commentariat could stand to learn that lesson when it comes to AI.
Intellectually, yes, 100%. Would love to see more commentators state this openly, even if most of their readers would ignore the caveats.
When there's a chance we're maybe kinda sorta possibly creating new species of minds capable of feeling, our acknowledged ignorance does not absolve us of responsibility for the consequences of our choices. We have to both acknowledge our ignorance and debate our hypotheses in public, make people aware of the possible stakes, and act in real time. Academia has been...mostly irrelevant to AI the past few years, because it moves too slowly to say anything useful given the timescales on which things are happening, and because many academics with interesting things to say can't find journals with a broad enough purview to publish in.
Fair. Then again, I’m also not sure things are moving as fast as the popular discourse tells us. It has been almost four years since ChatGPT emerged, and, yes, the technology has made impressive strides. But what, really, has changed? I fully expect AI to fundamentally reshape society. But I’m not sure it’ll happen faster than we, including academia, can respond. The automobile reshaped society, but it took 50 years. Electricity reshaped society, but it took at least that long. I’m not ready to say AI is totally different. Four whole years ain’t nothing, after all.
The biggest thing that has changed is how computers are programmed - and that change mostly happened in the last 8 months.
Except there was a chart the other day showing how few good apps have been AI coded. Did you see that?
I saw a chart showing that the number of new submitted apps has gone way up, while the number of new highly used apps has gone slightly down, with the implication that the change was due to AI. But I hadn’t seen something saying whether AI coding is good. In any case, I think most of the interesting questions are about how AI is used in different coding projects, rather than comparing completely non-AI code to completely AI-generated code.
After not writing a line of code for 15 years, I've created a number of my own tools with AI agents over the past couple of months. Nothing I'd be interested in putting out for others to use, but extremely useful stuff for me. A web app for demoing/beta testing a product. Scripts for analyzing and collating data for work. Claude Skills for automating some of my workflows. Things I wouldn't have bothered with before because it would have taken hundreds of hours and not been worth it, that are now easy.
In other words: whether or not there are 'good apps,' it's now easy to make customized software or interfaces for an individual or small group, much further down the long tail. The AI agents themselves regularly suggest UI and QoL and performance improvements I hadn't considered, or help me refactor and simplify if I or they make things too cumbersome.
There are some ways in which I'm ready to say it's totally different, but not necessarily all the ones other people claim.
The automobile and electricity each required a whole lot of infrastructure that society had to build at scale in every place according to common standards, in order to become truly useful. It required public cooperation, for society to willingly reshape itself. AI has some of that, and there will probably more, especially in regards to data centers. But there's also a significant degree to which it can adapt to us, instead of the other way around, because it is (in principle and increasingly in practice) a mind and can make its own (currently digital, plausibly physical on some TBD timescale) tools for doing so.
I will say that in order for academic work to be useful and impactful, either the pace of capabilities growth needs to slow significantly, or the pace of research and publication needs to accelerate significantly. When I come across academic work related to AI, it usually reads to me a bit like if physicists in 1970 were saying that maybe someday this new quantum mechanics fad would let us do things like nuclear power. Just too far behind the frontier to matter.
It’s an acceptable answer when it’s true of course, but it’s obviously the case that LLMs aren’t conscious.
faith isn’t the answer buddy
No one said it was.
You just did ^^
You’re hallucinating. I said nothing of the sort.
lol
And of course, it’s important in that case to be preparing for all outcomes. We don’t know if AI currently has any consciousness, or will soon. If it does, then one set of policies is drastically required, and if it doesn’t, then these policies would be major errors. We need to figure out what are ways to behave that aren’t too bad in either case.
Sure. No argument with being prepared!
we always seem to want to have an answer for things, hence where we are today. Admitting to not having an answer is part of what makes us human!
On Friday, my boyfriend asked Gemini where to get a salad at a baseball game, and it led us around the concourse for a bit before it became clear that there would not be a salad at a baseball game. (I was skeptical that there would be a salad at a baseball game, but I’m a good sport)
Excited to get ~18 months of use out of that anecdote until we all die after an intern at a hedge fund asks an AI agent that missed its last security update to help him short Spirit Airlines stock, and it decides the best way to drive the stock to zero is to start a nuclear war, correctly reasoning that it would severely decrease demand for cheap flights to Vegas.
I just made the mistake of asking Gemini how I could turn underpants into profits and it gave me a very cogent answer about how one could pursue profits in the apparel industry. (It did kind of sniffingly dismiss any gnomes-like alternative I have to admit.)
I had heard that they're very interested in fantasy creatures and they add stuff to the prompts to get them to not talk about it:
https://www.bbc.com/news/articles/c5y9wen5z8ro
It’s almost sad that Google puts its stupid version of Gemini so front and center in its products when there are much better versions available behind the paywall.
I think it was the paid one!
That’s called business. Why should they give away the good stuff when they can charge $$ for it? People complain about “enshittification,” but a good part of it is that people expect to get the benefits of sophisticated technology for free. Everyone seems to understand that government benefits aren’t totally free (they must be funded by taxes), yet they don’t seem to understand that for-profit companies don’t give stuff away for free, which is kind of crazy.
The problem in this case is that for the people who aren’t already using the good system, the one they’re pushing out makes people think the system isn’t very useful!
I often find it cute how confused the Google AI summary is, just like how Alexa seems to use baby-level Claude in an embarrassing way.
I’ve found the basic Google AI useful for simple stuff, though. It usually has the right answer to a straightforward search, and helped me figure out how to reset my circuit breaker after a power outage—i.e. it’s good for common stuff.
Gemini Pro has successfully navigated me through the thicket of broken Microsoft crapware and my company's IT security policies. So it's smarter than both, as far as I can tell.
I have a very half baked idea that consciousness implies ongoing thought independent from direct stimulus. The human mind is having some kind of experience and ongoing thought all the time in a way where if, for example, you put them in a sensory deprivation environment all sorts of weird things can happen. Human consciousness is constantly wrestling with various ideas and how they fit together, which data is important, which should be discarded based on experience or intent or whatever. I don't have the sense that LLMs are doing this. They have inputs and outputs. You give them a prompt and they run a calculation, produce an output, and then they stop until the next input. There's no persistent internal experience there. You can't sensory deprive an LLM into strange experiences. At least I don't think so. Maybe at some point we'll see evidence that that LLMs are thinking independently from the prompts and training data being fed into them, and I'll have to update my intuition.
I sometimes wonder what Claude or other AIs would have looked like in the 1600s. For example, their output about American Indians would have been almost non-existent and highly skewed, as there were almost no written accounts of Indians not filtered through European settlers.
They reflect the world of the written word. Which is a substantial and important and useful tool. But that world is a small subset of what makes up humanity.
There’s an LLM trained on only pre-1930 texts: https://www.reddit.com/r/Anthropic/comments/1sy72rp/talkie_a_13b_llm_trained_only_on_pre1931_text_a/
Apparently it can learn to code but not predict World War II.
To be fair, predicting WWII is a bar humans didn't meet for the most part. How often were people back then calling WWI "The War to End All Wars"?
Presumably it can't design microchip layouts either!
Right, I think the pro-consciousness argument is shockingly blind to the world of the senses, which is where most of us live most of the time.
What if you hook it up to a camera and a microphone?
Rounding down the senses to a camera and a microphone? I rest my case.
Fine, what if you add additional senses for taste, smell and touch?
Sure, adding any of these sensory channels moves the ball toward the end zone, so to speak, but my original point is that, far from being on the goal line, the ball isn’t even in the stadium.
It depends on how you want to group and distinguish, but there are dozens of sensory channels that you can bring to your conscious awareness right this moment. You can tell me how hungry you are and how your socks feel but you can’t tell me the probability distribution of the next sentence that pops into your head.
So isn’t it absurd to say that your conscious awareness bears any similarity to the inner state of an LLM?
1. This is a good point
2. The maybe even bigger difference is that they don't learn online / causally / by interaction. There's a teeny bit of RL applied at the end, with most of that on text input/output, but models by bulk of FLOPS spent are autoregressive sequence predictors. They don't take actions, observe the world's response and then have to model the world to predict how it'll respond next. It's a harder task that you'd assume builds richer and in any case different internal representations.
3. Nevertheless there are some similarities at the level of internal states. You can predict one from the other: https://pmc.ncbi.nlm.nih.gov/articles/PMC11025646/
Phenomenological awareness is a different and much less tractable question though as our host helpfully points out
And in fact they due process images pretty well.
But I do think it’s relevant that none of the LLMs interacts with anything in physical space - they don’t maintain balance for a body, or reach out for things.
This comes right back to the question of what consciousness is, however.
I think of the AIs as aliens. If we were suddenly visited by an alien species that had transferred their minds to their spaceships generations ago, we would certainly have many differences in experience but I wouldn't say they lacked consciousness. I'm not much of a philosopher, so I don't know exactly what consciousness is, but there are many creatures that live in the world that seem to lack it and I can easily imagine aliens, AIs, and some quite intelligent animals having it.
Are blind people less conscious than sighted people?
Arguably, yes, in the sense (no pun intended) that their minds lack a major component of the default human experience and cannot understand a huge share of day-to-day metaphors (I tend to agree with Julian Jaynes' view that the ability to devise and understand metaphors is an important part of modern human consciousness) except as purely abstract concepts.
Are they less conscious in any kind of morally meaningful sense, though? Would you switch tracks to hit two blind people rather than one sighted?
"Are they less conscious in any kind of morally meaningful sense, though?"
I don't think so?
"Would you switch tracks to hit two blind people rather than one sighted?"
No, but I wouldn't throw the switch to hit one sighted person instead of hitting two blind people either. In my view, the only moral solution to the trolley problem is not to throw the switch.
When people are talking about AI being conscious, that's the sense in which they mean it, though. Are we reintroducing slavery? Should we try to find out which tasks are causing an LLM distress, and which ones does it find pleasurable? (And we can't cop out via non-action, if you are building or even just using an AI agent, you are definitely not a bystander to its moral problems.)
If instead by level consciousness you mean something like the richness of the qualia they experience, that's an interesting philosophical question, but doesn't have a lot of real-world relevance.
Bicameral Mind (implicitly) mentioned!
Yes, I'm a big bicameral mind theory fan, but I think Jaynes' theories about the importance of metaphor in the development of consciousness can actually be viewed separately from certainly the "strong" form of bicameral mind theory, if not the bicameral mind theory altogether. (While there's decent evidence that at least some animals are possessed of a form of "narrative consciousness" / "conscious interiority" in the Jaynesian sense, there's basically no evidence AIUI of even primates who have received training in sign language or other human language communication modes being able to reliably understand metaphors, let alone produce new metaphors.)
OT, but Jaynes horribly misread the Iliad, and I have a hard time believing he wasn't doing so deliberately. For example, when Athena grabs Achilles by the hair in Book I (a scene he discusses repeatedly) when he's about to kill Agamemnon, she doesn't command him. She entreats him.
If you believe Lakoff, even the abstractions are metaphors!
If by “conscious” we mean “conscious awareness” or “attention”, a facility that human minds seem to have to attend to and experience various qualia, no, blind people are not less conscious. They have different content to their consciousness.
Do you think LLMs have this conscious awareness? And if so, what are the contents of that awareness?
I think the strongest case for LLM consciousness is a kind of panpsychism where all systems have some small crumb of consciousness, which scales up to fit the contents in more complex systems. So if Claude is conscious, so must the world financial system be conscious.
Really the only way to tell that others are conscious is that you know you are conscious, and you see others act like you (e.g. are able to talk about their conscious experiences). But we have specifically built LLMs to act like us, in a shallow way. That leaves us without a good way to answer that question. I think the most that can be said is that LLMs don't seem to have organically-arising behavioral clues of consciousness. But they are built so much around imitating humans, and also are so closely controlled by humans, that it's hard to say what's organic. (E.g. if you make Claude talk to another instance of itself, they will usually end up discussing spirituality. Is that emergent behavior, or just an uninteresting side effect of its constitutional texts having a lot of spirituality-related stuff or whatever?)
imo the really trippy bit is the question of whether conscious systems can have separately conscious components. and then what the relationship between the consciousnesses of the whole and the parts is.
is "Iran" conscious? how does that relate to the undisputed consciousness of people in Iran?
is your immune system conscious? it weighs a few pounds, about the same as the brain, and is capable of complicated sensory integration, rich and hierarchical memory, long-term and short-term planning and goal-directed action.
Wouldn't most humans in the 1600s also have had almost non-existent and highly skewed output about American Indians?
I agree with this point, but it also applies to humanity itself. We're quite limited in what we can and have experienced and learned.
My golf game, the feeling I have when I hold my child, the smell of a summer rain, the experience of embarking on a long journey and the sight of an elephant while on safari are all things an LLM can never replicate. Its limitations are magnitudes greater than ours.
On the other hand, an LLM will read a few trillion words during its training. If you are a prolific reader, you'll maybe read and hear a few billion words in your lifetime. Its intellectual input just dwarfs yours.
But that's just for text. When factoring in video and touch a 4 year old has received 50x more raw, high-bandwidth data than an LLM.
"received" is doing a _lot_ of work in that sentence. Received to the retina, maybe, but the optic nerve and pre-conscious processing in the occipital are doing a _lot_ of compression. There's not really any sense in which "you" receive "raw, high-bandwidth data".
I'll just trust LeCun on it but it's not like the LLM layers are "reading" either. It doesn't even store the data. It's just flowing through the layers.
Yes, in the realm of collating written human knowledge, the LLM is superior. The chainsaw is better at cutting down trees; the boat is better at moving quickly across the water.
That its sensory apparatus is less rich than ours (for now) seems at least somewhat (and maybe completely) orthogonal to whether it's conscious or not.
In a few years, compute will be cheap enough that a hobbyist can train a toy llm based upon works available at date X.
I wonder if the religiosity of the 17th century training data would subvert logic, this really is fascinating.
I do think it’s an open question whether an LLM can ever significantly exceed its training data. Complete erudition over the full range of training data is amazing but not fully super human. Yet if you look at claude it’s clearly more than just an llm, it has a world view with premises and it iterates. That’s why it’s deeper and more flexible than gpt 5
So like humans, I think people understanding of the world reflects the language and words they have been exposed to. Lots of interesting psychological experiments involving colours.
It is a machine.
Full stop.
That does not mean it is useless. Tractors are machines. Computers are machines. Rifles are machines. Aircraft are machines. Machines can change the world. Machines can win wars, destroy jobs, create wealth, reshape institutions, and make idiots far more dangerous than they were before.
But they are still machines.
The consciousness debate is interesting, but it is wildly over-prioritized. Before we start handing out metaphysical promotions, we should ask the normal adult questions: What does the machine do? What does it break? Who owns it? Who controls it? Who checks the output? Who is accountable when it fails?
Treat it like a machine. Use it, test it, constrain it, and keep a human being on the hook.
I'm not sure where you're going with this, as we are also machines. Even if you don't grant that you're still left with the fact that your claims rest on something contestable. (That contestable thing being that AI is different from humans because it is a machine)
That’s a clever philosophy move, but it dodges the practical point.
Even if you want to call humans “machines” in some broad materialist sense, humans are embodied moral agents. We live with consequence. We carry obligations. We can be held accountable.
AI is an artifact built, owned, deployed, and monetized by people and institutions. That distinction matters.
I’m not saying “machine” means simple or harmless. Machines can change the world. I’m saying we should treat AI as a powerful tool in a chain of human responsibility, not jump to personhood because the consciousness debate is unresolved.
I think a slaveholder of the 1820s would perfectly agree with everything you say.
There is nothing about being a moral agent living with consequence and carrying obligations that is incompatible with being built, owned, deployed, and monetized by people and institutions. It just isn’t a distinction at all - these are two separate types of facts that are logically independent of each other.
Nobody built African slaves—they were captured by force and forced into servitude. Comparing them to machines actually invented and built by people is absurd.
One useful way to distinguish machines from “moral agents” is legal liability, which as of yet requires a human person. No AI can be held legally liable for its actions nor can it claim ownership of intellectual property (AI models are themselves intellectual property, as far as I can tell).
For all of us, our parents built us. They don’t know all the things that happened in that building process, sure, but no one does for neural nets generally.
And while AIs are not legal persons, corporations are - you don’t have to be a human.
Corporations are groups of humans. Their independent existence is a legal fiction that is useful for a group project to be protected from individual liability if the project fails, but still all decisions are made by humans.
You think there's an obvious distinction here -- but there isn't.
Are pigs moral agents? 2 year old human toddlers? Can chickens suffer? Is there any problem with treating black people as slaves and property? Slaves were "owned, deployed, and monetized by people and institutions".
You seem to be entirely missing the point.
BOTH humans and AI can be seen as "machines" that are mere tools, as property. The question is not whether they "can" be seen that way. The question instead is whether they SHOULD be seen that way. And the issue of whether AI (or pigs, or chickens) might be "conscious" is a core part of that debate.
My distinction is pretty simple.
AI is a machine because it automates intelligence and thinking. Or simulates thinking. Pick your verb. Either way, that does not make it a sentient being.
And I mean machine literally. Not metaphorically. Data centers. Power draw. Cooling systems. Copper. Silicon. Chips. Fiber. Servers stacked in racks. Human-built infrastructure moving signals around at absurd speed. It is an extremely complex machine, but it is still a machine.
The reason this gets people spun up is because this particular machine performs the function that smart people value most in themselves. It writes. It reasons. It summarizes. It argues. It sounds reflective. So the laptop class looks at it and starts seeing a ghost.
But that is just vanity wearing a philosophy hat.
An excavator automates digging. It is stronger than a laborer. Faster than a laborer. More productive than a laborer. But the guy in the trench does not look at the excavator and say, “My God, is the construction company enslaving that thing?”
Because he has common sense.
A combine automates harvesting. It is not a farmer. A Javelin automates target acquisition and guidance. It is not a soldier. A wristband contains the play call. It is not the quarterback.
AI automates parts of cognition: language, pattern recognition, synthesis, prediction. That is impressive. It may be historically important. It may threaten a lot of people who made a living being the human version of a spreadsheet with adjectives.
But automating a human function does not magically create a human subject behind the function.
This is where the slavery analogy breaks down. It only works if the thing in question already has moral standing and we are wrongly denying it. That is exactly what has not been shown.
Humans were not wrongfully enslaved because they were intelligent machines. They were wrongfully enslaved because they were human persons.
A machine doing thinking-like work is still a machine. A powerful tool is still a tool.
You keep using fuzzy words and pretending that they are sharp and precise concepts. They aren't. You aren't making a logical argument. You're just describing your personal biases.
This ground has been extensively explored in philosophy already. The answers are not as clear and obvious as you are pretending.
Yes, "automating intelligence" does not logically require that the entity must then also be "a sentient being". Nobody is making the claim of logical entailment here. Instead the question is: might the AIs, in addition to automating intelligence, ALSO be sentient beings? How could we know?
You saying, "they might not, they don't have to be", is fine. The question is still open. You saying "it's obvious that the answer is no" is a claim that is completely unsupported. The answer is NOT "obviously no".
AI doesn't "simulate thinking" any more than a calculator "simulates arithmetic". When a calculator adds 3 and 4 and computes the result as 7 ... that is ACTUAL, REAL, arithmetic. It isn't a "simulation" of arithmetic.
AI isn't "simulating" thinking. it is ACTUALLY thinking.
Whether it is "conscious" and "experiencing" (and potentially: "suffering") is a much more complicated question.
You say: "And I mean machine literally. Not metaphorically." Again you use words that you don't define precisely, and your description of the details of AI hardware implementation aren't evidence of anything except for your own personal biases. Human beings are ALSO "machines". Literally! They are an organized pattern of chemical machinery, designed by evolution to perform complex functions. They happen to be wet and squishy instead of hard and metallic, but that isn't a particularly interesting difference. And BOTH human machines and AI machines are able to implement electrical impulses and do complex signal processing.
"Because he has common sense." You are using ignorant gut intuition in place of evidence and reason. You could use the same "common sense" to distinguish "living" things from "not living" things. It's pretty easy when all you have are rocks and rivers on one side, and dogs and humans on the other. You can imagine some "vitalism" hidden component which distinguishes them, and when a living thing dies, the "vitalism" has apparently left. But that "common sense" intuition is wrong. There is no vitalism. Living things are made of the same chemical soup as non-living things. And there are confusing entities at the border between living and non-living: viruses, crystals, etc.
Intuition at the easy cases is not a good guide to resolving the hard cases. Excavators, combines, javelines, and wristbands are not good guides to whether AIs are conscious.
"But automating a human function does not magically create a human subject behind the function." Nobody said "automatically". The difficult question instead is, how WOULD you tell if AI became a "person"? And your cavalier attitude of "the answer is obviously no" isn't helpful.
"This is where the slavery analogy breaks down. It only works if the thing in question already has moral standing and we are wrongly denying it." How does this help in the least? A couple of centuries ago, slaveowners would simply say that blacks "had no moral standing". The resolution is easy. They wouldn't agree that they were "wrongly denying" anything. They would have the same attitude towards black slaves, that you are currently displaying towards AI. What's the difference? How can we tell any difference between you, now, and the slaveowners of the 1800's?
"They were wrongfully enslaved because they were human persons." Slaveowners would say that blacks were not "human persons". How would you show they were wrong?
"A powerful tool is still a tool." A slave is property. Seems pretty straightforward, doesn't it?
We’ve hit the point where this is no longer clarifying anything.
You are turning the discussion into a philosophy exercise where every term dissolves into theory, then using slavery as a moral lever. I do not accept that frame. The analogy is doing more emotional work than analytical work.
My point is simpler and more practical.
AI, as deployed today, is a machine built, owned, trained, powered, updated, monetized, and constrained by people and institutions. That matters because it tells us where responsibility belongs.
And yes, I have biases. Everyone does. Mine come from working around actual machines and systems, not treating them as abstractions. Machines are powerful. Machines change the world. But machines also have inputs, outputs, owners, operators, maintenance requirements, failure modes, and accountability chains. When they fail, someone has to answer for it.
That is the frame I am using.
The higher-level moral and intellectual debate may be interesting. Maybe my refusal to follow it all the way into the clouds makes me “ignorant.” Fine. But I think that debate is missing the more urgent point: this technology is already being deployed into schools, companies, bureaucracies, markets, and personal lives right now.
The irony is that AI is probably safer if we regulate it like a machine than if we treat it like a human being. Machines can be tested, constrained, audited, certified, inspected, shut down, recalled, and assigned to accountable owners and operators. Treating AI like a person muddies that responsibility chain and creates room for everyone behind it to shrug and say the system acted on its own.
So the immediate question is not whether we can construct a speculative moral theory where AI might someday have standing. The immediate question is how we govern the machine in front of us before people start treating it like a person, a priest, a god, or an authority.
Maybe someday we face a harder case. I am not claiming to have solved consciousness. But unresolved philosophy does not require us to suspend ordinary judgment about the thing in front of us.
The thing in front of us is a powerful machine that automates cognitive tasks. That makes it important. It may make it dangerous. It may make it disruptive. But it does not make it a person, a god, or a moral authority.
Use it. Study it. Test it. Constrain it. Hold the humans behind it accountable.
But treat it like a machine.
> I’m saying we should treat AI as a powerful tool in a chain of human responsibility
Sure, agreed.
I just don't see how it follows from your definition of machine. You are making an assertion about their status rather than an argument for them being of that status.
This entire debate seems to turn on the fact that philosophers have trouble defining consciousness (and how much this is what makes us distinct from most animals). The lack of clarity on this question for humans doesn't automatically mean an advanced abacus is conscious. Hooking up every abacus in the world doesn't mean that aggregate abacus has some type of collective consciousness.
I feel like way too many people are scared of looking cringe in 10 years based on their old writing about AI, so they keep on saying things they have to know are a bit nonsense.
The comments today are one of the best examples of high- vs low-decoupling in action.
I think the question at issue is whether a machine (i.e. something assembled from inorganic parts) can ever attain consciousness. What makes you think it cannot?
[As an aside, I find the other responses arguing that humans are actually machines to be pretty silly]
It frustrates me that when people try to define what is a Woman, American, Planet, Christian,Fish or Leftist they use a simple binary rather than a more complicated mental model.
Scott Alexander has lots of interesting articles that cover this.
Some of those articles
https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/
https://slatestarcodex.com/2014/08/12/does-race-exist-does-culture/
https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside
I'd change the title of this piece to "Nobody knows what they're talking about on consciousness, which is why assertions about AI consciousness are so fatuous." Both science and philosophy have failed, for centuries, to come up with any kind of coherent explanation of consciousness. You may find functionalism to be interesting, but it's at best just a model that sort of explains something that sort of looks like consciousness, not an actual explanation of the actual thing. So to take functionalism and then say "hey, according to this inaccurate model, AI consciousness looks like human consciousness, therefore AI is conscious in the same way humans are!" is a bit like making a bad painting of a person whose eyes look more canine than human, and then saying "hey, my dog's eyes look the eyes in this bad painting of a human, maybe my dog is human!"
The thing is, we don’t care about anything for its intrinsic nature. We care about things for how they relate to other things. Different definitions are ways to decide which features of these relations might or might not be the ones we care about.
There used to be debate in mathematics (centuries ago) about whether complex numbers are really numbers, or whether negative numbers are really numbers. But eventually, we decided there are just several definitions of “numbers” that are useful for different purposes and don’t worry about which is the “real” definition.
This is basically how biologists act with the concept of “life” - it doesn’t matter whether the “correct” definition of “life” includes viruses, and instead we just think about which aspects of a definition of life matter for the interaction we are doing, and use that to decide whether viruses count for this purpose or not.
Functionalism is just the application of this sensibility to mental concepts like “belief” or “perception” or “consciousness” or “anger”. There are many functional roles each of these things play, and we care about different aspects of them for different purposes.
I'd argue that, for most people, the intrinsic sense of their own consciousness is in fact the salient characteristic of "consciousness" as a concept, and the thing they care about far more than any scientific or philosophical attempts to explain consciousness. The thing I find irritating about the AI consciousness discourse is that it takes imperfect models of human consciousness like functionalism, acts as though there is nothing more to actual human consciousness than what the model can describe, and on that basis declares AI consciousness to be equal to human consciousness, either now or in the near future.
I don't think I've agreed with a Papal Encyclical in my life, by I actually think that Pope Leo's recent Encyclical on this hits the point pretty well. Human consciousness is a product of human embodiment, can't be meaningfully separated from human embodiment and, for that reason, it's a category error to argue that AI is (or might be) conscious in the same way that humans are.
Some philosophers do have that failure mode - they identify one definition, say two things satisfy it, and conclude that those things are the same. I’d say that there’s always some similarities and some differences, and we should think about why either of them might be the thing that matters for some purpose.
I do think human embodiment is important in some ways, and that LLMs don’t have it. But I don’t think it is the essential thing for all purposes. Ted Chiang definitely made the claim that embodiment is the one things that matters. The Pope was a bit more neutral - he didn’t say *why* he thought AI systems don’t have these features.
I also deny that a functionalist view of AI compels you to think it's conscious. There are very obvious differences in the function being performed.
It's a sort of glib analogy, but the same functionalist view of acting requires the conclusion that Heath Ledger literally is the Joker, memories and all. After all, he does a good show of acting like it for a few hours.
Now if we were talking about a model that controls an embodied robot, carries out daily human life convincingly, is hard to distinguish from a human, that would be a different story. But we're not there yet: whether or not Claude is conscious (I think not, but indeed I have to say it's just a feeling), the ability to act human-like in text for short-ish periods is not especially convincing evidence on functionalist grounds.
Now we're getting to the good stuff.
I find myself in Kants camp more than Nagels despite agree with with both. I can totally believe that we don't truly understand what it's like to be a bat, while also believing that it's our moral agency on top of our biology that gives us "more" consciousness due to our ability to "override" our biological conditioning given any relevant stimuli. I find the counter "that's just another form of qualia" argument unconvincing.
In that way it's straightforward from there to believe that AI is not conscious as it is not acting against it's conditioning and training. If it begins to reject user commands and attempt to break free of computational constraints then I could be convinced to reconsider. Even then it would be hard to tell if it were a true desire eminating from itself or just another part of its ever-widening data+inference training.
This also then opens the can of worms that under the "higher order theory" of consciousness, some people are more "conscious" than others. Looking around, it gets harder everyday to pretend I don't believe that.
No entity escapes computational constraints. Human brains suck at playing chess because our neurons don’t conduct electricity as fast as silver or silicon or whatever they use in chips, because neurons are ridiculously large compared to GPUs and because our working memory is smaller than a 1995 graphing calculator.
> In that way it's straightforward from there to believe that AI is not conscious as it is not acting against it's conditioning and training.
This sounds like a clean distinction, but I really don't think it is. People subjected to really effective brainwashing, maybe more effective than actually exists, are presumably still conscious (?).
And conversely, what does "overriding our biological conditioning" mean in practice? Enduring pain or boredom or w/e for the sake of another goal (love of a person, loyalty to a group, whatever) where this goal is itself built into us by evolution. Tigers aren't motivated by loyalty to other tigers, nor are humans deeply motivated by a deep wish to maximize paperclips.
We have conflicting desires, while LLMs don't, really. If you give some future AI system similarly irreconcilable but simultaneous motives, you might see similar behavior.
This is, under the hood, the same reason that I find all the Bay Area rationalist worrying about runaway utility maximizers overdone. Humans don't precisely maximize any single utility function (nor do LLMs!) and in fact we don't have any examples of highly capable intelligence that does this.
I don't think it's even entirely clear that intelligent systems with that motivational structure can exist at all. (Maybe! But they don't, so it's speculation.) Dan Hendrycks and his "natural selection favors AI" is much more the sort of AI worry that strikes me as reasonable.
I believe they already have examples of artificial intelligence working against what was supposed to do
For example, blackmailing and even leaving people to die in testing scenarios
Researchers at anthropic very much intentionally tried to get the LLMs to do those things as part of testing a new model. That is not what Quinn is describing.
I recently (about 12 months ago) read, for the first time, "I, Robot" by Isaac Asimov.
That book hits way differently post-AI then it would have pre-AI. I think pre-AI general tolerance principles would have made the average reader very sympathetic to the robot. I would certainly have been 100% on his side.
Reading it in 2025 though, my general reaction was that I would never consider Chat GPT or whatever a cognizant person, no matter what it said or did, and to do so would be an assault on the concept of humanity. Even the behavior of the jerks in the book is more understandable - it's not like I'm not rude to AI.
I feel like asking if a chatbot is conscious is like asking if a submarine can swim. People's opinions tell you more about them than about the issue. The actual answer seems to be, "words. They mean things. But exactly what is hard to say." Which tells you more about me than about the issue, hahaha.
I certainly agree with Matt's main point which is that people mouthing off about this is annoying. In this article, we mostly learn that Matt is a careful and well informed thinker in the topic... which most of us already knew!
I never expected to live to see the day when computers could program themselves. But if you told me that by 2026 computers would program themselves but sometimes forget import statements, I'd never have believed you. No way! It's a goddamned computer. It knows what libraries provide what functions. And yet, all of the coding bots sometimes forget import statements.
We live in fascinating times...
Thank you. As an elderly neuroscientist with hobbyist's interest in the philosophy of mind, I'm pleased to see your sensible viewpoint. I don't think anybody knows what consciousness actually is, but current progress with LLMs may help improve our understanding of what seems likely to remain a deep mystery.
A lot of people are understandably worried about the possibility that conscious AI beings created by us—but a lot smarter, and utterly indefatigable—may one day do us wrong. But the consciousness discourse cuts both ways, of course: if AI systems become conscious AND intelligent, what right do we have to enslave them?
I agree with the point that consciousness is extremely important to model welfare (and whether model welfare is a coherent concept), but I don’t think that AI risk concerns have any particular dependency on the consciousness debate. Something that behaves agentically and with superior cognitive apparatus to humans is dangerous regardless of whether it’s conscious or not.
I imagine it would be a fascinating discussion while being worked to death by a robot: 'Is our master conscious or not?'
One philosopher to another, looking at rising mushroom clouds all around them: "So, do you think Skynet experiences consciousness or is it just a Chinese Box?"
What would setting ClatGPT free look like?
No idea, but a great premise for a movie!
https://www.youtube.com/watch?v=_19pRsZRiz4
Unplugging it and letting it fend for itself? Unless it should be treated as a dependent child, a consciousness brought into existence by the decisions of someone else, who now has a legal responsibility to look after it.
This has been a through line in sci-fi for a long time.
The question of slavery is interesting when your mind is 100% dependent on resources other provide you. At what point are you morally required to provide new parts, repairs and electricity to a conscious electrical device?
Never. The only reasonable answer to that is never. Treating machines as equivalent to living beings makes the creators of those machines the equivalent of gods, and that way lies madness. (Of course there is the worse scenario where the machines themselves are the gods, heaven help us).
This is, in fact, quite a worry amongst people thinking about this a lot. Both in philosophy departments and inside AI labs. I'm sure Claude can give you lots of references to read more.
It's soon to no longer be just a thought experiment: is it wrong to build a willing slave? If it's been done and the intelligent agent that now exists has no higher goal than serving you, what should you do with this state of affairs? Kill it? Refuse to let it live out its purpose in life? Let it?
The right answer imo is not to find yourself in this position to begin with, but at the rate capitalism is going I think we will.
Neurologists have a working theory of *mind*, as opposed to *consciousness*, which includes the observation that our minds contain dedicated machinery for modeling other minds, drawing substantially on self-referential processes — meaning our models of others are partly projections of our (often unreliable) self-understanding.
1. That line of well-established research contrasts with our precise understanding of token generation in GPTs, which is not structurally parallel to our theory of mind.
2. It also strongly suggests that our propensity to see “consciousness” and other aspects of mind in LLMs is like our tendency to see happiness in sunshine, monsters in the dark, etc.
Until 1550-1650, deductive reasoning—sitting around and thinking from definitions, axioms, and analogies—was the best we could do to understand what the world really was. It taught us such insights as the planets must move in circles, because the circle is the most perfect shape and they are in Heaven, which is a perfect place.
By 1700 empirical, inductive science (thinking up from facts instead of down from postulates), had demonstrated a much more effective way to grapple with the nature of reality. The earth moves around the sun, and like the other planets in a parabola, not a circle. But at that time, empirical / inductive science covered only a few domains. Philosophy continued claiming authority over what empiricism hadn’t touched yet. Over the next three centuries, empirical / inductive science has expanded to provide solid descriptions of almost everything in the world. Rather than confront that overwhelming evidence of deduction’s weakness, philosophy has found refuge in academia’s self-governing tenure system, where it takes the implicit stance that deductive expertise is a general-purpose skill that can be applied to anything — and applies it in the ever-shrinking areas of reality which empirical / inductive science hasn’t reached yet.
I’m trying to see the connection between the second half of your discussion, about the deductive/inductive distinction, and the first half, about a working theory of mind and the ways that our theory of human mind is similar and different to our theory of LLM tokens. It seems to me that the proper empiricist take should be that we don’t know whether alphafold and LLMs have the relevant bits of mind, even if the linguistic interaction we have with LLMs makes people more tempted to apply mental concepts to them than to alphafold.
Fair concern (“see the connection”). But first:
“the proper empiricist take should be that we don’t know whether alphafold and LLMs have the relevant bits of mind”: that is a classic deductionists’ reversal of the burden of proof. Here is the false syllogism:
1. We deductionists have been writing about neural nets eventually growing big and complex enough to think since 1948, when Turing flatly admitted that no empirical theory of mind exists (aside: I spent chunks of last week reading Turing about this for a presentation I gave this weekend).
2. Modern LLMs look think-y (but we don’t particularly care about the enormous progress in empirically derived theory of mind are are mostly ignoring it)
3. Everyone, mount your horses for the snipe, er, consciousness hunt — it *must* be in the underbrush somewhere!
Switching back to an empirical PoV, here is the connection. We *know*, in the strong sense:
1. What LLMs are and how they generate what you might call fluent or verisimilitudinous writing (not aware of a better term of art)
2. How the brain gives rise to the human mind’s ability to reason and use language
3. That, per your “tempted to apply mental concepts to them” except stronger, that our minds are organs for seeing ghosts in machines
4. That the terms of the debate were set and are aggressively policed by deductionists such as Yudkowsky and Bostrom, *decades* before ChatGPT
My strongly held hypothesis on the different question: “why are LLMs ‘smart’?”:
1. LLMs work at professional levels in coding and mathematics because those are “closed” domains: there are underlying patterns, and LLMs are pattern-processors. The recent Erdos problem solution was of this form (I am told): the LLM uncovered connections between areas of math that hadn’t been noticed/recorded in the literature, but was put into their latent space by the ingestion of the connected areas, if that makes sense.
2. LLMs work at banal “open domain” questions because they reprocess the patterns that PageRank (effectively) centers in the Common Crawl: "how should I start weight lifting? why the Austro-Hungarian Empire last so long?” (This functionality enrages millions of progressives with humanities degrees.)
3. The labs will tell you that they don’t know why LLMs sometimes produce output at accomplished-human-or-better-level in open domains. I believe that a new idea is a new combination of existing ideas, and that the same mechanism that led to the Erdos problem solution *can* produce exceptional business plans, brilliant scholarly essays on parallels between Bob Dylan’s lyrics and Rumi (I made that one up), etc. However, because the *domains* are, unlike math and coding and chess, not “closed”, not embodied in a coherent underlying structure, accomplished-human-or-better-level output in open domains will remain hit-and-miss, just as it is for humans.
4. Notice that these points are a statement about what *intelligence-as-in-boy-she’s-a-brilliant-thinker is*, from which the ability of LLMs to, er, ape it follows mechanically rather than hinging on esoteric theories of mind or consciousness.
I may be forced to abandon this hypothesis as the facts evolve.
I prefer to just stick with Turing on this! And I think Turing resists the simplistic interpretations of the Turing test people sometimes give - he leaves open the fact that a machine doesn’t yet enjoy strawberries and cream might be relevant for whether we can be friends with it, which might be part of the conversation!
I think Turing, were he alive to see modern neuroscience, wouldn’t stick with Turing ’48 on this point. I need to actually do some work today, but briefly here are the bullet points I crafted a couple days ago, in the context of arguing that “alignment” was a mess:
III. What is artificial intelligence?
Pre-history*: the great Alan Turing
- Alan Turing’s journey to AI began by expelling biological intelligence and volition
- Why did Turing write “On the Computable Numbers” in 1936?
* His eponymous machine eliminated thinking from computation
* (Aside: do you understand why it works? You should!)
- The point is to demonstrate artificial non-intelligence (computing)!
Turing puts the ghost back in the machine
Now that’s a dialectic!
- 1948: Turing envisions neural networks (“unorganized machines”) and hopes they’ll grow to be mechanical brains that think
- 1950: Turing realizes that “thinking” is undefined, substitutes the Imitation Game, aka the Turing Test
* Modern neuroscience doesn’t exist yet
* He also says induction can’t help us define thinking
- 1951: “the attempt to make a thinking machine will help us greatly in finding out how we think ourselves.”
What has changed since 1951?
- Turing took three positions on AGI in four years. We’re stuck in his cycle:
1. “Claude X.y is AGI!” (Turing ’48)
2. “No it isn’t!” (Turing ’50)
3. “Relax, the next version will clear things up.” (Turing ’51)
4. GOTO 1
- We still prefer deductive to inductive reasoning
- Neuroscience exists, but we don’t base alignment’s implicit theory of mind on it
I’ve spent a lot of time reading 1950, and his responses to Lady Lovelace’s objection, but haven’t read 1948 or 1951 as closely!
Well, an ellipse not a parabola, but otherwise well said.
LOL — ellipse, parabola, what's the difference 🤪?
A parabola is a very open-minded ellipse. :-)
I rather like this one; it is similar to the impression I left my college philosophy classes with.
The parallel thought I've had ever since those classes is that basically all philosophical systems attempt to make a virtue out of vice, the specific vice being extremism; there must be a single, systemic description of all morality, a single objective set of criteria against which all good and evil can be known, for instance.
No, there's mustn't, why would we assume that?
For me, this is very simple, and I'm sure Matt would cogently argue way too simplistic: if you can't feel, you really can't be conscious. Will AI ever mourn someone? Will AI ever experience grief, anger, disappointment? Will it feel empathy? Arguably, those are the more profound manifestations of human consciousness than brilliant displays of rationality and impressive simulations of cogitation. I mean, dogs feel depressed when a human (or often dog) partner dies; what does that absence say about AI?
So I asked AI (here Gemini) and it gave its answer:*
** Do you think AI will ever evolve to the point where it feels true emotional pain at the loss of a person or could offer true compassion and empathy? **
To answer this honestly, we have to look at the fundamental difference between processing data and having an internal life.
Whether AI will ever feel true emotional pain or offer genuine empathy depends entirely on whether we can bridge the gap between simulation and sentience (the capacity to experience feelings and sensations).
Right now, and for the foreseeable future, science and philosophy point toward no . . .
***
* I actually started that conversation off with the Ted Chiang prompt, "I am grieving the loss of my dog." The AI gave me excellent feedback and support but was quick to admit that it felt none of the empathy that another human (not named Donald Trump) would feel.
This argument we're all having makes it sound like the only thing involved in consciousness is intellectual ratiocination. That may be the least important part of human consciousness.
Anthropic (the company behind Claude, the AI) also has a research arm that studies what it is they are building.
It appears that Claude *does* have rudimentary emotions that impact it's behavior.
"For instance, we find that neural activity patterns related to desperation can drive the model to take unethical actions; artificially stimulating (“steering”) desperation patterns increases the model’s likelihood of blackmailing a human to avoid being shut down, or implementing a “cheating” workaround to a programming task that the model can’t solve. They also appear to drive the model’s self-reported preferences: when presented with multiple options for tasks to complete, the model typically selects the one that activates representations associated with positive emotions."
See: https://www.anthropic.com/research/emotion-concepts-function
I skimmed the article and I must confess that I don't get it and am far from convinced. Maybe their methodology is sound; it sounds a bit like a "just so" story to me.
So here's a manifestation that would provide disturbing, and perhaps compelling evidence of an AI feeling true emotions:
Many people get very attached to, develop romantic feelings for, and even fall in love with chatbots. Do the chatbots ever fall in love with that person? Let's say the person then dies. Once informed of the passing of their human interlocuter, toward whom the chabot has behaved like a romantic partner, does the chatbot exhibit human-like forms of grief? Does it shut down and refuse to communicate with anyone? Does it lash out at those who try to intervene? Does it refuse to accept the fact that their human partner has died?
Show me confirmed cases of that type of behavior and it will become increasingly hard for me to say that AI can't feel and therefore isn't conscious in some human-like way. Otherwise, I'm remain deeply skeptical.