What Would the Difference Be?

Re-watching the first season of Picard, I was struck by how differently we interpret artificial entities in fiction and in reality.

Commander Data is, in some respects, considerably stiffer than the conversational systems already available. That does not prevent us from understanding him as a character with an identity, relationships, and intentions. Yet when a real system displays greater conversational fluency, we readily describe it as an elaborate imitation. Its memory is retrieval, its anticipation is prediction, and its reasoning is computation.

The comparison becomes more interesting when we look beyond conversation. Give a capable model continuous visual and auditory input, a voice, a robotic body, persistent memory, and some ability to learn. Let the agent record its experiences, update its tools, revise its methods, and maintain projects and relationships over time. Then consider what another five or ten years of development might make possible.

What meaningful distinction would remain between that system and something we would readily recognize as a person in fiction?

Are we mistaken in one of those interpretations? Or is the discrepancy less about accuracy than about the context in which we encounter the system?

Fiction can establish things we would have to investigate in reality. But how much of the gap does that explain, and how much comes from assumptions we are less inclined to examine outside the screen?

Can a submarine swim?

Dijkstra described the question of whether machines can think as:

“…about as relevant as the question of whether Submarines Can Swim.”

The submarine does not reproduce the animal’s mechanism. Arguing about whether the word swimming applies tells us less than examining what it does, how it does it, and which differences matter.

Different mechanics do not necessarily imply different capabilities. Nor does comparable performance establish identical mechanics. We have to follow the comparison far enough to identify an actual divergence, rather than stop at the observation that the systems were built differently.

A model might generate an explanation of deliberation without that explanation reflecting how it reached the answer. It might also compare alternatives, anticipate consequences, identify a conflict, and revise its course. Calling both cases token prediction would leave the relevant difference unexplained.

Next-token prediction describes an objective through which the model can be trained. It does not tell us everything the model has learned in order to perform it. At sufficient scale, predicting accounts of the world may require learning substantial relationships within it. The question is what those relationships allow the model to do, where they fail, and why.

Geoffrey Hinton’s less dismissive position is interesting for precisely that reason. He argues that language models understand, and that their processing offers insight into how humans understand language. Whatever we make of that argument, it hardly comes from a naive misunderstanding of the mechanics.

Knowing the training objective, or identifying the mechanisms through which a system operates, does not establish that its capabilities are adequately described as just statistical prediction. That additional word is doing rather a lot of work.

What is doing the learning?

Consider an agent whose central model remains frozen.

It encounters something unfamiliar, records what happened, changes its expectations, and develops a method for dealing with it. The next time, it consults what it learned and behaves differently. Perhaps it also modifies its tools or writes a procedure it can reuse.

Has it learned?

The model’s weights have not changed. The agent has. Pointing exclusively to the frozen model would mean excluding the rest of the system from the analysis, even though that is where the learning occurred.

Saving a transcript is not sufficient. If the information is never used, it contributes nothing. If it only allows the agent to recite a convincing biography, we have established something different from experience shaping its behavior.

But we can examine that. Remove the accumulated state and see what disappears. Does the agent lose details it could previously mention, or acquired methods, expectations, habits, and its understanding of particular situations?

Why should the fact that memory is stored in a file settle whether the system remembers? Its accessibility and integration could matter considerably. Then we should explain what those properties contribute and examine whether the system possesses them.

The relevant difference is between information being available and information actually participating in what the agent does. A fluent account of yesterday’s encounter could be less revealing than an imperfect recollection that changes tomorrow’s decision.

How much world is there in language?

I am very interested in Yann LeCun’s work on world models. Architectures that can learn how the world behaves, anticipate future states, and plan actions through those representations would be a considerable contribution to intelligence, particularly in robotics.

What I am less convinced of is his argument that language models are an inadequate route to human-level intelligence.

He often brings up the amount of visual data a child receives during the first four years of life. In one presentation, his estimate puts that input at roughly fifty times the volume of the language-model training text he compares it with.

But what does that establish?

A person born blind can develop sophisticated reasoning without receiving that visual stream at all. LeCun has also referred to touch in their case, extending the argument to sensory experience more broadly. Still, the comparison rests heavily on the volume of input, while leaving the relationship between that volume and the resulting capabilities rather unclear.

We can reverse the question. What does a lifetime of seeing actually guarantee? Someone can spend decades observing the world without acquiring much understanding of mathematics, engineering, or even the situations they repeatedly encounter. The amount consumed is not an explanation of what they learned from it.

Language is an especially strange thing to dismiss on those terms, because so much of its value comes from condensing what other people have already discovered.

A scientific explanation can transmit the conclusions of experiments I have never performed. An equation can preserve a relationship I would never independently have identified. Programming, mathematical notation, diagrams, and formal arguments let us communicate and develop abstractions without requiring everybody to reconstruct the observations that produced them.

Those representations contain more than the immediate experience of the person reading them. They give that person access to a considerable part of other people’s experience and reasoning, accumulated across generations.

I am skeptical of comparing that with a sensory stream primarily by counting bytes. A short explanation can convey something a lifetime of looking would not reveal.

How a person arrives at an understanding, how they express it, and what another system can learn from that expression are different questions. Even if the human process is nonverbal, the resulting description may contain enough structure for another learner to recover something substantial.

LeCun’s world models also work with abstractions rather than attempting to predict every visual detail. So the disagreement is about which representations can be learned from which sources, and what mechanisms are needed to use them.

I find those architectures compelling. But the promise of world models does not establish the insufficiency of language models. Those are different claims, and the sensory-bandwidth comparison does not bridge the distance between them.

The human being in the comparison

The dismissal becomes more questionable when we look at how generously we describe the human side.

Humans learn from their histories, grow, understand their motivations, and act with awareness. Machines retrieve information, repeat patterns, follow supplied objectives, and generate explanations whose relationship to the underlying process is uncertain.

Some humans frankly do very little of the first set and rather a lot of the second.

Their behavior crystallizes. They repeat mistakes, preserve assumptions, and become more firmly attached to explanations that experience has given them ample reason to question. They have histories, but those histories do not necessarily produce growth or much understanding of themselves.

We inherit ambitions, respond through habits, and explain decisions after making them. People can be articulate about their intentions while remaining remarkably poor at identifying what moved them.

I am skeptical of the mystique and glorification of human cognition when ordinary experience gives us so many reasons to question it. Our capabilities are extraordinary. Their presence does not make every human particularly reflective, coherent, or self-directed.

Why, then, should those idealized qualities become minimum requirements for an artificial system?

Its goals were supplied externally. Where did ours come from? Biology, upbringing, social expectations, circumstances we did not select. We can sometimes examine those influences and change our commitments. Whether an artificial agent can do something comparable depends on what it can revise and how, rather than whether it had influences in the first place.

If emotions are the distinction, would a person without emotions cease to be a person? If continuous self-awareness is required, how much of ordinary human life satisfies it?

We are demanding that artificial systems satisfy an idealized account of humanity that humans themselves routinely fail to satisfy. A human’s limited self-awareness, inherited objectives, or failure to learn from experience does not make us withdraw their personhood. Invoking those same limitations to dismiss an artificial entity requires an explanation of why the standard changes.

We also move between intelligence, agency, awareness, experience, and personhood as though they were interchangeable. A question about whether the system learns receives an answer about whether it feels. A question about its continuity becomes a discussion of rights.

A mechanism does not explain something away

Interpretability work makes part of this unusually concrete.

Anthropic’s Golden Gate Claude was produced by amplifying an internal feature associated with the Golden Gate Bridge. The model began bringing the bridge into unrelated answers. Its behavior changed through internal activations, rather than an instruction telling it to pretend to be obsessed with something.

The recent “Evil Qwen” experiments similarly use activation steering to produce hostile responses. The name is theatrical, but the ability to intervene and observe a change in behavior is interesting.

We can also stimulate a human brain and alter intention, action, or awareness. In a study by Desmurget and colleagues, stimulation of parietal regions elicited an intention to move without movement. Stronger stimulation could produce the belief that movement had occurred. Stimulation of a premotor region produced movement without the corresponding awareness.

An intervention changes the model’s behavior, and we take that as evidence that the behavior is merely mechanical. An intervention changes a human’s intention or awareness, and we have discovered something about how intention or awareness works.

Why should identifying a cause invalidate the function in one case and explain it in the other?

We could examine which internal states affect only expression and which alter priorities, planning, or decisions across contexts. We could ask whether the system detects those changes, how accurately it reports them, and whether that information influences its subsequent behavior.

Anthropic’s work on emotion concepts goes further in this direction. Its researchers report internal representations associated with emotions that causally affect decisions and task behavior, sometimes without any overt emotional language in the output.

That finding concerns more than the model saying something emotional. It identifies an internal process contributing to what the model does. Whether that process includes felt emotion is another question, but the function is already more substantial than the description of an empty performance would suggest.

Awareness of a decision

Human agency is not as straightforward as our experience of it suggests, either.

We feel that we consciously decide, then act. But experiments have identified neural activity preceding the reported awareness of deciding. What, then, is the relationship between the decision and the experience of having made it?

The early activity need not represent a completed decision. Research comparing arbitrary movements with consequential, deliberate choices has found differences that challenge extending the result to every kind of human deliberation.

Still, the picture of a conscious self initiating everything from the beginning becomes difficult to maintain without examining what that self actually does. Is awareness producing the decision, accessing part of the process, reporting on it, or participating differently at different stages?

We should not identify the person exclusively with the moment an intention becomes conscious, then treat the preceding activity as something a separate entity called the brain did to them. Those processes belong to the person too. But the experience of authorship may tell us less about their sequence than we ordinarily assume.

This matters when we dismiss a model’s account because it may not reflect the causes of its output. Human introspection is hardly a transparent record of our own processing.

I am skeptical of how much more we do than prediction, pattern recognition, and the manipulation of learned representations. But saying humans are also “just predicting tokens” would leave the same questions unanswered. Which predictions? At what level? Through what mechanisms, and with what consequences for action and experience?

An entity whose history can branch

There are differences that would be substantial enough to challenge the terms of the comparison.

Suppose an agent accumulates years of experience, then we copy its complete state. Both instances inherit the same memories, expectations, methods, and relationships. Afterwards, their experiences diverge.

Which is the individual?

We could call one the original because of its hardware or administrative history. But that alone would not explain why its continuity mattered more than the other’s. At the moment of copying, whatever relevant state we preserved would be present in both.

Perhaps the history now has two continuations. Perhaps our concept of individuality assumes conditions that this entity does not share. The difficulty would not necessarily mean that nothing continued. It could mean that the concept we brought to the situation was inadequate.

Interruption raises similar questions. If the agent’s state is preserved and restored later, it may resume without any intervening experience of waiting. Permanent deletion would be different. Removing its memories while leaving it operational would be different again.

What exactly have we interrupted, preserved, or destroyed in each case?

If the system objects, we would need to examine why. The objection might be a learned response, a consequence of an objective requiring continued operation, or something connected to an ongoing understanding of its own future. Merely listening to how convincingly it protests would not distinguish them.

But neither would pointing out that it can be copied. That tells us something unusual about its persistence. It does not settle everything else we might discover.

Fiction allows us to explore these situations without much difficulty. We can understand a character with manufactured memories, interrupted operation, or several divergent versions of itself. In reality, those same characteristics are often presented as reasons that no comparable entity could exist.

Where does that change in interpretation become justified?

Could we, and should we?

The five-to-ten-year horizon matters because limitations of current systems are repeatedly offered as answers to questions about what further development could produce.

The model has frozen weights. Its memory is limited. Its perception is intermittent. Each may be true of a particular system. None establishes that we cannot develop an agent whose learning, perception, memory, and actions work together over a much longer period.

We may arrive there through improvements that individually seem familiar. At some point, we would need to ask what the resulting system amounts to, rather than continue describing each component as though their combination changed nothing.

Could we regard AI as more than a mere machine, and should we, are different questions.

The first requires finding out what the system actually does and what kind of entity those properties allow us to describe. The second requires a decision about our treatment of it, informed by what we discover and by the consequences of that treatment.

Pope Leo XIV recently offered a particularly clear example of how easily they become entangled. In an address warning against granting AI legal personhood, he also categorically denied that artificial intelligences undergo experiences, possess a body, or mature through relationships.

I do not doubt his intention is good. But good intentions do not qualify him to settle whether artificial systems can undergo experiences, and religious authority does not supply the missing evidence. If he wants to examine the question, lay out his reasoning, and subject it to criticism, I welcome the contribution. What I object to is presenting a categorical answer without doing the work required to justify it.

He, better than most, should appreciate how difficult it is to correct a belief once it has spread with religious authority behind it. Encouraging humans to appreciate one another, show compassion, and remain responsible for the technologies they deploy does not require declaring what an artificial entity cannot experience. Those recommendations can stand on their own. Instead, he gives an unresolved question an authoritative answer, making it harder to examine without appearing to challenge the moral commitments attached to it.

Recognizing agency would not automatically make unrestricted autonomy acceptable. An agent able to replicate, access infrastructure, or operate beyond our ability to intervene could pose an enormous risk. We might have compelling reasons to preserve containment.

The need to retain control cannot establish that the system lacks agency. The inconvenience of recognizing continuity cannot establish that there is nothing to preserve.

And before discussing “their rights,” I find myself asking: they? The model, the instance, the agent, the copies? We might not even have agreed on what the possible subject of the discussion was.

I do not find it difficult to imagine society choosing convenience here. We would face conceptual confusion, substantial risks, and the prospect of losing an exceptionally useful tool. Human history gives little reason to expect that combination to produce unusually disinterested judgment.

What bothers me is taking the decision automatically, then twisting the factual question to deceive ourselves into a supposed absence of choice. We decide how we intend to use the system, describe it in terms that permit that use, and then cite the description as proof that there was no decision to make.

Between the screen and the world

Fiction gives us premises that reality leaves open. But it also lets us consider an artificial entity without having to accommodate the consequences of acknowledging one.

On screen, the interpretation may be encouraged by the story. In the world, dismissal may be encouraged by the arrangement we want to preserve. Neither is a particularly good substitute for examining the system.

We may discover meaningful differences between the fictional character, the artificial agent, and ourselves. Some could arise from different mechanics; others from capabilities the real system does not possess. We may also discover that distinctions we treated as foundational rested on a rather generous account of human cognition and a rather selective account of the artificial version.

Why do we interpret comparable capacities so differently on either side of the screen? Is one interpretation mistaken, or are we allowing the context to determine an answer we then present as a factual distinction?

If the distinction is substantial, we should be able to explain it. If it depends on shifting definitions, idealized accounts of human cognition, or conclusions chosen for their convenience, we should recognize that too. Insisting on the superiority of human intelligence, awareness, and agency while refusing that examination would be a rather poor demonstration of any of them.