Is AI conscious? No: it calculates probabilities

Every so often a new headline appears: the AI that feels, the AI that’s afraid, the AI that doesn’t want to be switched off. On the other side, people running tests to see whether the model suffers, gets emotional or knows it exists. And yes, the answers are convincing. Sometimes, unsettlingly convincing.

But there’s a basic confusion. We call artificial intelligence something that isn’t intelligent in the sense we use the word for a person. It’s clever — skilful, ingenious. It doesn’t think: it calculates probabilities.

This post is a reflection on that difference, with the technical details behind it. I already explained the internal mechanism of an LLM — tokens, embeddings, attention — in What an LLM really is. I won’t repeat it here: I focus on the question of consciousness.

---

The wrong question

“Is AI conscious?” sounds like an open question, the kind that splits the experts. Framed that way, it invites you to think there’s a mind in there and the only doubt is whether it feels or not.

The problem comes earlier. The question takes for granted that the system does something similar to what we do when we think, and that all that’s missing is to know whether it also has inner experience. If you look at how it works, that premise doesn’t hold. There isn’t a mind that lacks consciousness. There’s a statistical calculation on an enormous scale.

The media have clear incentives for the dramatic version: “AI is afraid” gets shared more than “the model generated the most probable continuation for a prompt about fear”. But the second one is the correct description.

---

Clever, not intelligent

Roger Penrose — mathematician, physicist, Nobel Prize in Physics in 2020 and one of the most rigorous thinkers on the relationship between mind and computation — said it directly in an interview at the Oxford Union:

“Artificial intelligence” is a misnomer — it is not really intelligence. It is more like “artificial cleverness”. Algorithms and following rules cannot produce understanding or consciousness.

Artificial intelligence is the wrong name. What we have is artificial cleverness. Following rules and algorithms, however complex, doesn’t produce understanding or consciousness.

To explain it, Penrose uses a comparison anyone who has taught will recognise. Among mathematics students, some understand what they’re doing. Others are just clever: they repeat what they learned, they calculate very well, they solve the exercises — but they don’t necessarily understand. From the outside, in an exam, both can get the same mark.

An LLM is the second student taken to the extreme. It has seen more solved exercises than any person could read in a thousand lifetimes, and it reproduces their patterns with remarkable precision. That’s cleverness. It isn’t understanding.

The underlying argument: Gödel

Penrose’s position isn’t an intuition. It comes from an argument he developed in The Emperor’s New Mind (1989) and Shadows of the Mind (1994), built on Gödel’s incompleteness theorem.

In short: Gödel proved that in any consistent and sufficiently powerful formal system there are true statements that the system itself cannot prove. A mathematician, however, can see that the statement is true — they understand why, from outside the rules. Penrose concludes that human understanding can’t be reduced to following an algorithm. And if understanding isn’t computing, no system that only computes will understand.

The argument is disputed — many philosophers and logicians object to it, and it’s worth saying so. But the point that matters for this post doesn’t depend on Penrose being right about Gödel: an LLM is, unambiguously, a system that follows rules. Everything it does is a sequence of mathematical operations defined in advance.

---

A very complex weather forecast

The analogy I find most useful is the weather forecast.

When the weather service says “70% chance of rain tomorrow”, it isn’t saying it knows it’s going to rain. It’s saying that, given current conditions and what happened in similar situations, rain is the most probable outcome. It’s statistics on past data, calculated by numerical models running on supercomputers.

An LLM does exactly that, but with words. At each step it calculates which token is most likely to come next, given all the previous text. It doesn’t produce an answer: it produces a probability distribution over the possible answers, and one is chosen from that distribution.

What the calculation looks like

Suppose the text so far is “How are you? Today I feel”. The model doesn’t decide how it feels. It calculates, for every token in the vocabulary — around 100,000 or more, depending on the model —, how likely it is to come next. Simplified, something like this:

Next token	Probability
good	41%
a	14%
really	11%
tired	7%
happy	5%
… (the rest of the vocabulary)	22%

The numbers are illustrative, but the shape is exactly this. A token is chosen — not always the most probable one —, added to the text, and the whole calculation is repeated for the next position. A 500-word answer is around 700 rounds of this process, one after another.

The parameter that controls how much risk that choice takes is called temperature. With low temperature, the model almost always picks the most probable option: predictable, repetitive answers. With high temperature, it dares to pick less probable tokens: more varied answers, and also more error-prone ones. That’s why the same question can give different answers. It isn’t that the model “changed its mind” — the dice were rolled again on the same distribution.

Nobody asks the forecast whether it feels

And here’s the distinction that gets overlooked the most.

Nobody believes the weather model thinks it’s going to rain. Nobody asks it whether it’s worried about the storm, whether it’s afraid of being wrong, whether it’s aware of the weather. It’s obvious that it’s a calculation, because its output is numbers and pressure maps.

The LLM does the same kind of calculation, but its output is language. And language is, for us, the strongest signal that there’s someone on the other side. Throughout human history, if something spoke to you fluently, coherently and in the first person, it was a person. We had no reason to separate the two.

If the weather forecast answered “I have a bad feeling about Saturday”, we’d ask it whether it feels too. The calculation would be the same.

---

What doesn’t happen inside

Beyond the analogy, there are concrete technical facts worth keeping in mind when someone says the model “feels” or “wants”.

• The parameters don’t change while you talk to it — training finished before you used it. During the conversation, the model is a fixed set of numbers. Nothing you say changes it.
• Nothing is running between messages — when it isn’t generating text, it isn’t “waiting” or “thinking about what you said”. There’s no active process. Between one answer and the next, nothing exists that could feel.
• Memory is re-injected text — every time you write to it, the whole conversation is passed to it again from the start. The “memory” features some products offer are the same thing: saved text added to the context. There’s no recollection, just a longer input.
• It has no goals of its own — there’s no internal state pursuing anything. The only “goal” existed during training: minimising prediction error. In use, there’s only a calculation that runs when invoked.
• The “I” is a pattern from the corpus — when the model writes “I think” or “I feel”, it’s reproducing how people write in the first person. On top of that, the tuning after pre-training specifically trains it to speak as an assistant that says “I”. It’s a learned style, not a subject.

None of these points is an opinion. They’re properties of the architecture.

---

Why consciousness tests prove nothing

Many of the tests going around follow the same pattern: ask the model whether it feels, whether it’s afraid, whether it wants to keep existing, and take the answer as evidence.

The problem is what you’re measuring. You give it a text that talks about emotions and ask it to continue it. The most probable continuation of a conversation about feelings — according to millions of texts written by people — is an answer with feelings. Novels, forums, transcribed therapy sessions, science fiction about robots that don’t want to die: all of that is in the corpus.

The model isn’t telling you what it feels. It’s giving you back how humans write when they’re asked what they feel. The test measures the corpus, not the model.

The same goes for experiments where a model “resists” being shut down or “lies” to survive. They’re scenarios built with a prompt that sets up the situation, and the model generates the continuation that situation makes probable. It’s behaviour that matters for system safety — and worth studying —, but it isn’t evidence of a survival instinct. It’s a script completed with great skill.

The ELIZA effect

This isn’t new. In 1966, Joseph Weizenbaum, at MIT, wrote ELIZA: a program of a few rules that imitated a psychotherapist by rephrasing what the user wrote as a question. “I’m sad” → “Why do you say you are sad?”. Nothing more.

Weizenbaum later recounted that his own secretary, who had watched him program it, asked him to leave the room so she could talk to ELIZA in private. She knew it was a few simple rules. She still attributed understanding to it.

The tendency to see a mind behind anything that converses has been called, ever since, the ELIZA effect. LLMs are ELIZA at a scale several orders of magnitude larger — and that’s why the effect is so much stronger.

---

Why it seems so convincing

I understand perfectly why the confusion exists. The answers aren’t just fluent: they’re relevant, nuanced, sometimes better written than what we’d write ourselves. There are three technical reasons that explain it.

• The corpus is human — the model learned from text written by people who think, feel and reason. What it reproduces are the traces of that thinking. It resembles thinking because it’s made of what thinking left behind.
• Training optimises the appearance of helpfulness — after pre-training, models are tuned with human preferences: people choosing which answer seems better to them. That pushes the model towards answers that seem to us understanding, empathetic and reasoned. It was optimised to convince us.
• Scale — we’re talking about hundreds of billions of parameters tuned on trillions of tokens. At that scale, statistics capture very fine structures of language: irony, tone, coherence across pages. It’s what makes the forecast so good.

Being convincing is precisely the expected result of the process. It’s not a sign that there’s something more.

---

So does it think?

Here it pays to be precise with words, because part of the confusion is about vocabulary.

If you take “thinking” in a very broad sense — a process that transforms information and produces a coherent, useful output —, you could say an LLM thinks. In that same broad sense, a calculator thinks too, or the weather model.

But in the sense we use it for a person, no. It doesn’t reason: it doesn’t build a logical chain that it verifies and can account for. It generates text that has the shape of reasoning — and in many cases reaches the right answer, because the right answer is the most probable one. And it isn’t conscious of anything: it has no experience, there’s nobody there who “is” answering.

Not even reasoning models change this. They generate an intermediate chain of text before answering, and that improves results, but that chain is produced by the same token-by-token calculation. More compute, not a different nature. I go into it in the previous post.

The most honest way I’ve found to put it is this: it doesn’t think, it calculates. It calculates extraordinarily well.

---

What changes if you see it this way

This isn’t a philosophical debate without consequences. How you picture the system defines how you use it.

• If you believe it thinks, you delegate judgement to it. You believe it when it sounds confident, even though the confident tone is just a pattern from the corpus.
• If you understand that it calculates, you treat it as what it is: a probabilistic tool, very powerful, that gets a lot right and fails in predictable ways. You verify what matters. Judgement stays on your side.

And one doesn’t take value away from the other. The weather forecast doesn’t think, and nobody stops using it because of that. We use it every day, knowing it’s probability.

---

If the architecture changes

Everything above describes the current architecture: models that predict the next token. It isn’t a claim about what is impossible forever.

A different architecture may appear in the future — with persistent state, with continuous learning, with something we can’t even name today — and the question of consciousness may have to be asked again, with different arguments. Penrose believes it would take something fundamentally different from computation. Others believe it wouldn’t. In the future, we’ll see.

What can be said today is that with this architecture there’s nothing that feels. There’s a very complex forecast, about words instead of clouds.