The World It Is Entering

One of the issues with AI is the mismatch between the technology and the world it is entering. At times, it feels as though we had given nuclear weapons to medieval men. We are becoming extraordinarily capable without necessarily becoming much better qualified to decide what to do with those capabilities.

That includes the people developing them.

Technology is rarely produced by a representative sample of the society that will receive it. It is disproportionately developed by small groups with unusual intellectual abilities, specialized knowledge, access to resources, or some combination of those things. They can see possibilities others cannot, or pursue possibilities others cannot afford to pursue.

Those advantages do not necessarily make them wiser, more culturally mature, or better qualified to administer the consequences of their inventions. Being able to build something does not imply a particularly good understanding of the people who will have to live with it.

I do not think this is an arrangement we could simply replace with a more representative process of invention. Unusual people pursuing things that seem unnecessary, implausible, or outright ridiculous are part of how progress happens. Sometimes the population cannot express demand for something because it has no experience from which to imagine it.

But that leaves an interval between what a few people have become capable of producing and what the rest of society is prepared to absorb. AI is making that interval rather difficult to ignore.

The time that investment buys

A technology can be useful before there is a viable business around it. The necessary infrastructure may not exist. Potential customers may not understand what it offers, or their organizations may be arranged in ways that make adopting it difficult. The product itself may need years of work before its usefulness justifies its cost.

Investment buffers that mismatch. It allows development to continue while the surrounding world acquires some of what is needed to make the technology practical. Some demand only becomes possible after somebody has done the work; waiting for a sufficiently informed population to request every invention would leave rather a lot uninvented.

The problem is what happens to that allowance of time once it comes with a valuation, competitors, successive rounds, and expectations about the return.

The company receives money so it can survive the interval, then finds itself under considerable pressure to make the interval disappear. Adoption has to accelerate. Customers have to reorganize. A capability becomes an inevitability, then an obligation: if you are not already using it, you are falling behind.

What began as a way to accommodate the distance between development and demand can end up increasing the distance between development and preparedness. The financing provides time for adoption while creating incentives to force it, manufacture demand, and impose urgency on people whose readiness has not advanced along with the product.

I explored much of that investor culture in “Who Made This a Race?” But the underlying mismatch would remain even under considerably more patient financing. A slower company would still be introducing something into a society whose institutions, habits, and expectations were formed without it.

Money can keep the company alive while that changes. It cannot purchase the corresponding cultural development on the same terms as more compute.

Progress does not wait politely

Society cannot become fully prepared in advance, either. Technology often helps produce the cultural changes needed to accommodate it. People encounter new possibilities, develop expectations around them, discover consequences they had not anticipated, and begin arguing about what should be permitted or preserved. Some of that understanding requires experience.

Nor is cultural adaptation simply a matter of teaching everyone how to operate the new product. It involves deciding what we want from it, what we are willing to give up, and what should remain possible for people who do not want it.

A workplace may become more productive while becoming less pleasant to inhabit. A service may become cheaper while becoming effectively inaccessible to somebody who needs to speak to a person. An institution can adopt the technology successfully according to its own measurements while transferring substantial difficulty to everybody dealing with it.

Those are not necessarily failures of technical understanding. People can understand what is happening and disagree about whether it is desirable.

This is part of why I find the language of inevitable progress so irritating. It takes a process full of decisions and presents the consequences as though they had arrived with the weather.

Still, a consequential invention changes the conditions under which people live. Even a responsible introduction would involve disruption, disagreement, and arrangements that turn out not to work. Some friction may be inseparable from the process through which society develops the capacity to accommodate it.

Accepting that does not require accepting every injury as an unavoidable price of progress. It does mean that technological and cultural development cannot be made perfectly synchronous. One often creates the conditions that propel the other.

With AI, the concern is how much faster technological development can become while those cultural processes still require time.

The technology joins its own development

Computers have helped us build better computers for a long time. Scientific instruments have enabled discoveries that made better instruments possible. Technology contributing to its own advancement is not new.

What changes with AI is the range of work it can contribute.

The system can help write the software, examine the literature, propose experiments, analyze results, and increasingly participate in the research through which its successors are developed. We do not need a model autonomously training a better version of itself for that feedback to matter. Partial automation can already shorten the process.

The labs are quite explicit about pursuing this. OpenAI describes developing an automated researcher; Anthropic reports substantial AI participation in its own research and development, including tasks the model leads under human supervision.

Coding and mathematics are therefore more than attractive products or impressive demonstrations. Improvements in those capabilities can improve the machinery through which further improvements are obtained. A lab trying to remain competitive has considerable reason to prioritize them. Under the financing conditions already described, that reason becomes rather urgent.

But the feedback has an uncomfortable asymmetry. The research organization can use the new capability immediately to help produce the next one. The society receiving it still has to educate people, revise practices, negotiate interests, and establish some workable understanding of what just changed.

AI can assist that work, but much of it still depends on human participation and experience. Its pace cannot simply follow the pace of the research.

If the time needed to produce a new capability falls faster than the time needed to absorb it, the mismatch ceases to be a transitional delay. It becomes a condition that technological development continually intensifies.

Even the people leading

The acceleration also complicates the assumption that the developers themselves are keeping up.

The usual account places them ahead and everybody else behind. The experts understand the technology; the public and the government need to catch up. But knowing how to train a system, knowing what it can reliably do, understanding how it reaches a particular result, and anticipating what will happen when millions of people use it are not the same accomplishment.

A lab can be exceptionally competent at the first while still investigating the others.

The recent mathematical results make part of this unusually visible. Producing a proof, checking it formally, explaining why it works, and understanding what follows from it are related but distinct activities. OpenAI is organizing collaborations with mathematicians specifically to examine and explain results its systems have produced.

What interests me is the possibility of that explanatory work accumulating faster than the people qualified to do it can proceed. A system may produce another result while the community is still digesting the last one. Eventually, even the most informed participants could find themselves working within a body of knowledge they increasingly depend on other systems to navigate.

Independent verification can remain possible while becoming more demanding, more selective, and more dependent on the tools being examined. The existence of a method of oversight does not tell us how much of the relevant work people will actually be able to oversee.

The nuclear comparison helps distinguish these problems. The scientists developing nuclear weapons understood the physics sufficiently to build them; that did not qualify them, or the governments directing them, to manage every geopolitical consequence. Technical mastery and responsible use were already separate questions.

AI introduces the prospect of an additional gap within technical mastery itself. The people directing the research could retain authority while becoming increasingly dependent on the system to explain which directions are possible, promising, or safe. Their decisions would increasingly rely on an understanding supplied by the thing they are supposed to direct.

Being qualified to control it

There is something rather convenient about discussing AI risk almost entirely as a question of whether the machine obeys us. Obedience leaves us with the qualifications of the person giving the instructions.

The intelligence required to construct a powerful system does not automatically supply the judgment required to choose its objectives. Financial success does not supply it either. Neither does cultural prestige or winning an election.

Our political institutions are supposed to reconcile different kinds of knowledge and competing interests. In practice, lobbying, campaigning, concentrated resources, and the difficulty of becoming properly informed all influence whose judgment becomes a decision. The people with the greatest ability to shape deployment are not necessarily the people with the best understanding of its consequences.

Giving them more capable instruments can enlarge the consequences of getting it wrong.

This is why I do not think we need an extinction scenario before taking the mismatch seriously. Employment, public administration, education, security, and the conditions under which people form beliefs are substantial enough concerns already. A society can retain control of its technology while using it in ways it is poorly equipped to manage.

And as the technology begins contributing to its own development, that inadequacy affects more than its immediate applications. It affects our ability to decide how to proceed with the technology itself.

Can we accelerate the other side?

AI could help us adapt through education, legal analysis, public administration, and the explanation of complicated systems. It could improve our understanding of AI itself.

But producing a good explanation is different from somebody acquiring an understanding. Producing a policy proposal is different from establishing whether it serves the people affected, persuading them, implementing it, and discovering what was missed.

Some of those processes are slow for bad reasons. Others take time because people have to participate, develop competence, or reconcile interests that genuinely conflict.

We could automate more of the decision-making too. That would accelerate decisions. Whether it accelerated our capacity to govern them would remain a separate question.

There is even the possibility of improving performance while weakening the understanding needed for oversight. If people increasingly delegate the work through which they used to acquire expertise, an institution may become more capable of producing an answer and less capable of examining it.

Closing the gap therefore requires more than making powerful systems available to the institutions falling behind. It requires developing human and institutional capacities whose progress is harder to measure, harder to sell, and less convenient to demonstrate in the next funding round.

It also requires recognizing how much development remains necessary. We are rather accustomed to treating our civilization as the established framework within which new capabilities must find their place. I am less convinced that the framework itself is particularly mature.

We may live in the past

When we imagine history, the past is enormous and the future is strangely small.

Behind us are civilizations, revolutions, entire ways of organizing life. We can distinguish periods by their architecture, institutions, beliefs, and understanding of the world. Ahead of us, the picture often becomes a vague extension of the present: better equipment, unfamiliar buildings, perhaps a disaster. We remain recognizably ourselves, facing recognizably our problems.

We place ourselves at the very forefront of history, which is true only in the rather uninteresting sense that nobody can inhabit a time that has not happened yet. It does not tell us how far civilization has developed, how much remains possible, or how our moment might look from somewhere considerably further along.

I think we actually live in the past. We may even live in what a future civilization would consider its prehistory.

Someone living thousands of years ago could also have looked at their society’s accumulated knowledge and reasonably felt that an extraordinary amount had been accomplished. They would have been right. Their mistake would have been assuming that this gave them much understanding of how much remained.

I find it difficult to look at our institutions, our treatment of one another, or the things we still struggle to understand and conclude that we are anywhere close to a mature civilization. There is an enormous amount of room for cultural development, quite apart from what remains scientifically and technologically possible.

Even when we expect radical change, we often preserve the categories through which we already understand the world. We imagine transformed employment, transformed markets, transformed government. It is harder to imagine that some of those concepts might eventually become as inadequate as older ways of organizing life now seem to us.

From that perspective, AI is more than a futuristic technology entering an established world. It is something an early civilization is bringing into existence, with implications extending considerably beyond the arrangements that civilization currently understands.

We tend to reserve primitiveness for other people’s centuries.

A character in the story

Our understanding of those implications is also limited by the assumption that we remain the protagonists.

History, as we usually encounter it, leads toward humanity and then becomes increasingly concerned with us. We are human, and we are choosing what to describe. But history itself has no obligation to preserve the subject of the books we write about it.

We have not existed since the beginning, and we have no assurance that we will remain the principal agents of everything that follows. Being responsible for an extraordinary development does not guarantee that we will remain at its center.

AI makes that possibility less abstract. A system that can formulate research questions, produce discoveries, develop successors, and organize substantial parts of that work would participate in determining what happens next. Its questions, recommendations, and discoveries would influence which purposes we adopt, not merely how efficiently we pursue them.

We tend to ask how such systems will change our world. Implicitly, we remain in place, receiving the changes. But the relationship itself could change. Humanity might become one participant in a process it initiated and no longer predominantly directs.

We could become part of arrangements in which the distinctions between a person, an institution, and a technological system no longer work quite as they do now. We could be foundational and still be early; enormously consequential without remaining the most consequential participants throughout the development that follows.

That possibility changes what I understand the mismatch to involve. We are judging the technology through institutions and assumptions that may be rudimentary, while also assuming a permanent role for ourselves that history has never guaranteed.

What we are developing

Most of what we call technology is something we construct to extend our abilities. Even when it transforms society, we tend to imagine ourselves providing the purposes and setting the direction. We build it, introduce it, and eventually establish a workable relationship with it.

Something that increasingly contributes to its own development could make that relationship difficult to maintain in its familiar form. The people directing it would be relying on it to help determine what comes next, while the society expected to accommodate it would have progressively less time to understand what had already happened.

Full recursive improvement would intensify that process. But it does not have to arrive before the mismatch becomes consequential. The feedback already exists.

I do not think we can currently envision what a mature version of this technology would represent, much less the place we would occupy alongside it. It may be substantially different from the kind of instrument we still imagine ourselves developing.

We are approaching it with considerable urgency, through institutions, habits, and assumptions that may themselves be remarkably early attempts at organizing a civilization. That is what makes the situation unsettling to me: the possibility that we are much earlier than we think, developing something whose consequences extend beyond our understanding, while remaining accustomed to treating ourselves as the mature party in the encounter.