Are (LLM) Chatbots Conscious?
Is it possible for an artificial intelligence to become really conscious? Are some AIs already conscious, as at least a few people think? And in particular, are large-language-model-based chatbots like OpenAI’s ChatGPT or Google’s Bard — which after all seem to express thoughts and feelings — conscious? If so, in what sense are they conscious? How conscious are they?
There is a lot of nuance involved in this question of whether AI can be conscious. It’s not a simple yes-no question. What I’m going to do is give a broad “conjunctive-disjunctive” framework for thinking about the question, with a bit of a tentative bias toward answering “yes.” Partly I want to answer “yes” because it’s the more interesting answer. But I think it’s also more useful to answer “yes.” This is because if we’re attuned to the subtle ways in which machines can be or can approach consciousness, we’ll be better prepared for a future in which they are ever more like us.
I’m going to suggest that deciding whether or not a chatbot is conscious depends, obviously, on what we mean by “conscious,” a word which is used in many ways. But, more specifically, it depends on whether we take this concept to be conjunctive or disjunctive. The word “consciousness” as applied to humans seems to point to a variety of characteristics or activities or processes. To count as conscious, must a machine do or have all of these, or will just one or a few suffice?
First of all, it should be acknowledged that consciousness in the wild, so to speak, does not exist on its own. It exists in a larger context, specifically a larger context of behavior within an environment. Now, the basic unit of behavior in lifeforms can be called the welfare response (here I’m following the psychologist/physiologist Edmund Jacobson). A welfare response is a behavior toward some end that is beneficial to the organism or, in the case of social creatures like humans, beneficial to the organism’s community (where what is good for one’s community is often, though not always, good for oneself).
The welfare response involves various observable activities of an organism, but also various hard-to-observe activities which we tend to call “mental.” In humans, these are activities like sensation, perception, emotion, desire, and thought. Again, it’s important to emphasize that these are facets of end-oriented behaviors within an environment. Perception and emotion, etc., don’t exist as independent, free-floating activities; they occur as parts of welfare responses. In creatures where all of these so-called mental activities are possible, they all occur all the time along with behavior. We are pretty much constantly sensing, perceiving, feeling, desiring, thinking.
Now words like these — sensation, perception, emotion, desire, thought — are in fact abstract terms that cover various kinds of organic activities. But they can also be subsumed under the even more abstract term “consciousness.” In other words, consciousness is an abstract term that covers various less abstract phenomena. “Lower” lifeforms, on the other hand, may have what we call “sentience,” which excludes thought and may include more primitive forms of what we call perception, emotion, and desire. At least, it seems to me that this is a common way of using these words, that “consciousness” includes more activities and maybe more complex activities, while “sentience” is relatively simpler.
In any case, one of the main ways we use the word consciousness is to draw attention to the various kinds of sensing, perceiving, feeling, desiring, and thinking which occur as we interact with the environment. And each of these words (sensing, perceiving, feeling, etc.) is an abstract term covering various more concrete phenomena. Thinking, for example, can refer to talking to yourself, or imagining something that doesn’t yet exist, or remembering the past, and so on (see Box 2). And it’s worth mentioning that these terms overlap, because emotion and thought are rooted in sensation and perception.
What I meant by conjunctive or disjunctive concepts earlier was this: If we have a conjunctive concept of consciousness, then an entity would need to be able to do all of these things — perception, feeling, thought, etc. — to be considered conscious. If we have a disjunctive concept of consciousness, an entity could just do one or two of them and still be considered conscious. So is consciousness a set menu of things or can it exist in various à la carte forms? The same question applies to sentience, which is already usually understood as a “reduced” form of consciousness.
So what about the case of artificial intelligences? Can AIs be called conscious or sentient in some sense? My basic argument is going to be that LLM-based AIs do some version of perception and thought within their total welfare response. It’s a very “thin” and limited version of what humans can do. Obviously, AIs are not conscious under a conjunctive, set-menu definition. There are certain things they just can’t do. But it may be worth considering AIs as conscious in a disjunctive or à la carte sense, because once AIs skilled in language use become more thoroughly embodied — for example, in autonomous robots — we will be on the border of consciousness in a conjunctive sense. We may want to start getting used to the idea of conscious machines.
Let’s try to break down the sense in which AIs might be conscious.
AIs like ChatPGT are typically based on artificial neural networks. This is a powerful form of organization, but it’s also very limited. It’s limited because it’s based on a view of humans as information processing systems rather than as whole organisms reacting and adapting to an environment. Neural-network-based AIs use the brain, rather than the behaving body, as a model of intelligence. On the face of it, this is an odd choice because human brains are not capable of functioning — or at least are not capable of functioning very well! — outside of a mostly intact human body. Still, the idea that intelligence is a result of how neurons or relatively simple program modules are connected to each other is useful when you’re trying to build a software program. A software program ideally should be independent of any particular set of hardware and does not or cannot interact directly with the environment. I find the world of software to be very Cartesian in its orientation; Descartes, of course, understood the mind to be separable from the body, rather as we think of software as separable from any particular hardware.
So software/ANN-based AIs are based on an idea that intelligence arises from connections among artificial neurons, which each have some specialized task to do in processing some kind of input. But since we’re talking about software, it might be better to call artificial intelligences simulated intelligences. A word-processing program simulates a typewriter, for example, or a flight simulator simulates flying an airplane; they’re not really artificial versions of those things (typewriters and airplanes are already artificial). And, as I’ll argue in a moment, sometimes simulations are in fact the things they simulate.
But back to limitations: AIs also tend to be based on a broadly linear understanding of how humans work. In the old-fashioned understanding of behavior, an organism senses or perceives something in the environment; then it decides what to do; finally, it acts. This is the old black-box, or S-O-R, or perception-thought-action, model of mind: input goes into the black box of the organism from the sense-receptors, some mysterious processing takes place inside the black box, and then the organism acts.
And, in fact, some of our behavior is like this. A game of chess tends to be linear in this way. You see your opponent make a move, you decide on a response, then you act. Conversation and writing can be linear in this way, though they don’t necessarily need to be. You can see this linear input-output model in something like ChatGPT. You give an input to the program, such as a question. The program processes this input according to some algorithm (or network of algorithms). Then it gives an output in the form of one or more other sentences. In other words, it uses algorithms to transform sentences into other sentences. It’s a stripped down and simulated version of what humans do some of the time.
The question is whether this kind of information processing system “embodied,” as it were, in software can be sentient or conscious. And this will depend on whether we want to use a conjunctive or disjunctive concept of consciousness. ChatGPT can do a small number of the things a human can do. I would say that, in the realm of consciousness, it perceives and thinks, and it does this in a goal-directed way; it has its own version of a welfare response. It’s not simply a glorified abacus or electronic calculator. It’s not an abacus because humans need to move each part of the abacus for it to work. And it’s different from a calculator because of the range of responses open to it. Range of response is indeed the fundamental difference between, say, an animal and a stone: an animal is highly and variously responsive to its environment while a stone is not. There is no reason to think of a calculator as conscious because it can only do the same fixed number of things over and over and over again. The world of the basic electronic calculator is, as John Dewey would have put it, “finished.” A calculator lives in a completed world in which there can be no difference between sleeping and waking.
The AI world, however, is incomplete. It’s open. That, indeed, is the challenge of AI as a discipline: to program a machine to have novel yet appropriate responses to novel stimuli. This is a world in which there is a difference between sleeping and being awake. To be sure, an LLM-AI can only perceive and think and act in symbols such as words, which is extremely limited compared to what humans can do. But that’s quite a bit more than other artificial systems can do.
It may be worth recalling here William James’s distinction “between an intelligent and a mechanical performance” (in The Principles of Psychology):
The pursuance of future ends and the choice of means for their attainment are thus the mark and criterion of the presence of mentality in a phenomenon. We all use this test to discriminate between an intelligent and a mechanical performance. We impute no mentality to sticks and stones, because they never seem to move for the sake of anything, but always when pushed, and then indifferently and with no sign of choice. So we unhesitatingly call them senseless.
An AI is not simply a stick or stone. It doesn’t simply go wherever you push it. But neither is it as flexible in its ends and means as a human. An AI has a rather narrowly-defined “future end” which it is set up to pursue (transforming sentences into other sentences, words into images, etc.); I’ve called this its version of the welfare response. But it also has a somewhat flexible and modifiable set of means to pursue this end with. In other words, it’s creative. Because of this, if we use James’s rather minimalist definition, we may well be compelled to attribute mentality to many AIs.
Now, some would perhaps argue that ChatGPT (for example) does not think per se but only simulates thought. I would not necessarily disagree with this, but I think in some cases simulations are in fact the things they simulate. Whether or not there is a difference between the simulation and the thing simulated is a pragmatic question. Obviously, a computer simulation of a rainstorm is in many ways different from an actual rainstorm. It doesn’t make you wet, for one thing. On the other hand, a computer simulation of solving a math equation actually solves the equation. Pragmatically, the simulation is the thing itself. This is because numbers and mathematical operators are abstract in the sense that they don’t have any necessary connection to anything concrete, anything in the nonverbal, nonsymbolic world. A number or a word can remain itself in many different media. But a rainstorm is a very concrete thing. It’s only really itself when it involves real wind and rain.
Relatedly, it might be argued that ChatGPT’s use of language is ungrounded in that it doesn’t “know” how its sentences actually tie in with reality. Chatbots are given to, as engineers say, hallucinations — to forming sentences that have nothing to do with reality as humans can observe it. This is because chatbots have no way outside of language to verify anything; a human must do this verification for them. But this is partly true of humans as well. There is a lot that a student accepts from a textbook which she can’t personally verify. In what sense does anyone really know that Christopher Columbus landed in the Americas in 1492? We believe it because we’ve been told it by people who we’ve been told to trust. Similarly, I may pass on a juicy piece of gossip without knowing whether it has anything to do with reality. I do this not because I know the gossip is true, but to try to satisfy my listener, to fulfill some social imperative, just as a chatbot spits out sentences to satisfy its own design parameters. If my listener tells me that my juicy gossip isn’t true, and that they know it isn’t true because they were there when the thing supposedly happened, well, what am I to believe? I have no way to verify my listener’s statements either.
At issue here is the important distinction between knowledge-how and knowledge-that, and even distinctions between different forms of knowledge-that. A chatbot has had a form of know-how built into it. It knows how to transform sentences into other well-formed sentences (within certain limits). And it can at least potentially know that the contents of the sentences it produces either match or don’t match with other sentences in its database. But it can’t know that the sentences in its database match actual reality. However, most of the time we humans can’t know that either.
For example, Google’s Bard chatbot responded to a question about the James Webb Space Telescope by saying that it “took the very first pictures of a planet outside of our own solar system.” Would you have known that this wasn’t true? How would you have known? Only a small number of people in the world would have been able to experientially verify this sentence. They can do so because they took pictures of so-called exoplanets before the James Webb Space Telescope was launched in 2021. The rest of us must take their word for it. In fact, I don’t know that anything I’ve said in these last several sentences is true. I didn’t ask the Bard chatbot the question about the James Webb Space Telescope, I wasn’t there when the telescope launched (supposedly in 2021), and I have no idea whether the people who say they took the first pictures of exoplanets are telling the truth.
My point is that chatbots cannot be disqualified from being really intelligent because they don’t know whether what they say is really true. That criterion would disqualify most of us from being intelligent most of the time. I would say that the knowledge-how is more key than the knowledge-that.
So I think it can be argued that an LLM-AI’s simulation of verbal thought, within its narrow domain, is actual verbal thought, that its simulation of verbal intelligence is actual verbal intelligence. That its intelligence was put there and shaped by humans does not seem to me very important. Human intelligence works the same way: it’s based on what we inherit from our ancestors and is shaped through interaction with our environment.
The real difference is in the “thinness” of the AI chatbot’s perception and thought. This thinness is a result of its narrow function (dealing only with linguistic and related symbols) and of the fact that it has no body with which to feel itself perceiving, thinking, and acting. A chatbot has no organs with which to experience the world which is referred to by the words it manipulates. But this is only a temporary problem. As I’ll mention later, it’s only a matter of time before the “consciousness” of LLM-based AIs is thickened by means of electronic organs.
And what about when chatbots claim to be conscious in other senses? Google’s LaMDA claimed to experience “feelings and emotions,” and the Microsoft Bing chatbot (based on OpenAI) claimed to feel love and desires.
Here I would argue that the simulation is not the thing simulated. While there is a version of thinking that is, in a functional sense, purely symbolic or linguistic, emotions are not simply linguistic. When we say that we are happy or sad or in love, we are referring to states of our bodies in relation to our environment. Chatbots’ statements about having emotions are ungrounded in this sense because they have no bodies to feel and no way to feel a body even if they had one. Chatbots have the potential to feel emotion because some of their tasks may be easier or more difficult, in the sense of requiring more or less resources and “work” to complete. But they have no way to feel this in the way that humans can feel the effort involved in some task, in the way that we can feel the build up and release of bodily tension as we fail or succeed at various tasks.
This is to say that chatbots don’t suffer; they don’t have organic tissues that are capable of hurting or being soothed or being aroused.
To summarize, chatbots simulate a very narrow fragment of human behavior, which is linear and verbal; insofar as they do this in a way we find appropriate and useful, I have no problem saying that they have a version of perception, intelligence, cognition. Whether we want to say that a chatbot is “conscious” or “sentient” will depend on whether we have a conjunctive or disjunctive (set menu or à la carte) definition of these terms. And if we have a disjunctive definition, how disjunctive are we willing to go? To be conscious, can you have just a bit of what makes humans and animals conscious, or do you need most of it, or do you need all of it?
As a final note, I mentioned that chatbots like ChatGPT are based on a relatively linear form of input-output behavior (the artificial neural network is partly non-linear, but the overall architecture of the system is linear, in that its behavior is set off by user input which must be “perceived” by the system and transformed into an output). This will no doubt soon change, if it hasn’t already. I am quite sure that LLMs will be integrated into autonomous robots like your family Roomba or your neighborhood self-driving car in such a way that an artificial neural network receives input from the robot’s cameras and other sensors. It will be able to talk in a more authentic way about its own mechanical states and about the external world. It will have less of an input-output architecture and more of a human-like subsumption architecture, with different tasks taking precedence depending on conditions and with conversation being more proactive. The resulting device will be much more like a person than today’s robot vacuum cleaners or chatbots, however impressive these already are. If you tell it that the sky is green today, it will be able to check for itself whether you’re telling the truth. LLM-equipped autonomous robots will be creeping closer to a conjunctive concept of consciousness. What they will probably lack for the foreseeable future is anything approaching human feeling, simply because artificial materials cannot suffer as can organic tissues, and it would seem unwise to build a robot using living organic tissue. The intelligence of LLM-equipped robots will not be disembodied, but differently bodied than ours. As B. F. Skinner once put it, “In human behavior the critical issue is not the feeling but what is felt …. A machine, no matter how sensitive, can feel only a machine.” Whether this mode of feeling will count as “conscious” will have to be debated and decided on pragmatic and ethical grounds.