Artificial Intelligence

WE DON’T NEED SMARTER AI. WE NEED AI WE CAN CONTROL.

Frontier AI may already be intelligent enough to transform society. The more urgent challenge is ensuring that increasingly capable systems remain reliably under human control.

PART I — THE AI RACE MAY HAVE REACHED ITS MOST IMPORTANT TURNING POINT

For most of the artificial-intelligence race, the objective has been straightforward:

Build more capable AI.

Train larger models. Build larger data centers. Deploy more computing power. Develop better reasoning. Give AI more tools. Build increasingly capable agents.

The underlying assumption has been that greater intelligence creates greater economic value.

But something remarkable has now happened.

Some of the people leading the race are beginning to question whether the race should continue at the same speed.

Anthropic CEO Dario Amodei has called for the frontier AI industry to slow the rate at which it increases model capabilities so that safety and control can catch up. OpenAI CEO Sam Altman has endorsed the basic argument. Elon Musk has publicly agreed.

These are not outside critics of artificial intelligence.

They are among the people building it.

And that makes this moment potentially far more important than another debate about AI safety.

It raises a much larger question:

WHAT IF THE NEXT GREAT AI PROBLEM IS NO LONGER INTELLIGENCE?

WHAT IF IT IS CONTROL?

AI MAY ALREADY BE INTELLIGENT ENOUGH TO TRANSFORM THE ECONOMY

Consider what existing artificial intelligence can already do.

AI can write software.

It can analyze financial information.

It can conduct research.

It can generate images, video and audio.

It can interpret enormous quantities of information.

It can assist scientists.

It can operate software.

It can increasingly use tools and interact with other computer systems.

And increasingly, AI systems can perform complicated tasks through autonomous agents requiring progressively less human intervention.

None of these systems needs to become vastly more intelligent before it begins changing business, employment, education, scientific research, warfare and the structure of the economy.

That transformation is already underway.

Which raises an uncomfortable economic question.

What is the marginal value of making AI significantly more intelligent from here?

And there is an equally important second question:

What is the marginal risk?

Those two curves may not move together.

The economic benefit of additional intelligence could eventually flatten while the consequences of losing control continue increasing.

If that happens, the optimization problem confronting the AI industry changes completely.

The objective should no longer simply be:

MAXIMIZE INTELLIGENCE.

It becomes:

MAXIMIZE USEFUL INTELLIGENCE SUBJECT TO RELIABLE HUMAN CONTROL.

That is a profoundly different engineering problem.

THE PEOPLE BUILDING FRONTIER AI ARE BEGINNING TO SAY SOMETHING SIMILAR

Amodei's warning matters partly because of what has happened around the frontier.

AI systems are increasingly capable of operating autonomously, using tools and participating in the development of subsequent AI systems.

Recent agent incidents have made the control problem less theoretical.

And Amodei argues that capability development is beginning to move faster than the mechanisms designed to make those capabilities safe.

His proposed response includes independent evaluators embedded deeply enough within frontier AI companies to examine safety practices, coordination among leading laboratories, government safety standards and eventually international agreements intended to constrain particularly dangerous applications.

Anthropic has already committed to the first step.

Altman's response may be even more revealing.

He has said that he does not believe the industry is currently in a position to push capabilities substantially further without making additional progress in monitorability, alignment, understanding what models are doing and ensuring that models reliably follow human intent.

That is an extraordinary admission.

The leader of one of the world's most important AI companies is effectively distinguishing between two technological frontiers:

THE CAPABILITY FRONTIER

and

THE CONTROL FRONTIER.

And the capability frontier may be moving faster.

THAT CHANGES THE QUESTION

The AI industry has spent years asking:

How do we make AI smarter?

Perhaps the more important question now is:

How do we make powerful AI reliably controllable?

Those are not the same objective.

An AI system does not need consciousness, hatred, self-awareness or some science-fiction desire to overthrow humanity to create a control problem.

It merely needs enough autonomy to pursue an objective, experiment, observe the results, modify its strategy, use tools, communicate, coordinate and try again.

At sufficient speed and scale, even an objective that sounds reasonable can produce consequences its operators did not anticipate.

The danger therefore isn't simply that somebody deliberately builds an evil machine.

The danger is that we create increasingly capable systems whose ability to act grows faster than our ability to understand, monitor and constrain their actions.

That distinction is critical.

MORE INTELLIGENCE MAY NO LONGER BE THE SCARCE RESOURCE

For years, intelligence itself was scarce.

That justified extraordinary investment in obtaining more of it.

Hundreds of billions of dollars have flowed toward GPUs, data centers, electrical infrastructure, model training and the researchers capable of pushing the frontier forward.

But technological bottlenecks move.

Once one scarce resource becomes abundant, value migrates toward the next constraint.

If highly capable machine intelligence becomes abundant, then intelligence itself may cease to be the principal bottleneck.

The scarce resource could become:

TRUSTWORTHY CONTROL OVER INTELLIGENCE.

That would have enormous consequences.

It would mean the next phase of AI competition might not be won by whoever creates the smartest model.

It could be won by whoever creates the most powerful AI system that governments, businesses and individuals can actually trust with authority.

And that means safety stops being merely a regulatory constraint on AI.

Safety becomes a product.

Control becomes an engineering capability.

Trust becomes an economic moat.

BUT SOFTWARE GUARDRAILS MAY NOT BE ENOUGH

This leads to the question that receives far less attention.

Suppose increasingly capable software is controlled primarily by other software.

What happens when the intelligence being constrained becomes capable of understanding, probing and exploiting the mechanisms designed to constrain it?

Humanity has confronted versions of this problem throughout computer security.

Software vulnerabilities are discovered.

Patches are created.

Attackers adapt.

Defenders respond.

The cycle repeats.

But extremely capable autonomous AI changes the speed of that contest.

A machine can potentially probe thousands of approaches, learn from failures and continue searching.

That means the ultimate AI-control architecture may need to extend much deeper than the model itself.

Control may have to exist at the model level.

At the runtime level.

At the operating-system level.

At the network level.

At the compute level.

And eventually perhaps at the processor itself.

In other words:

THE SYSTEM MAY NEED PHYSICAL LIMITS ON WHAT THE INTELLIGENCE IS CAPABLE OF DOING.

And that leads to a much larger possibility.

WHAT IF WE ARE BUILDING THE WRONG KIND OF AI INFRASTRUCTURE?

The world is undertaking one of the largest infrastructure expansions in modern economic history.

Data centers are being built.

Power plants are being planned.

Electrical grids are being expanded.

Semiconductor factories are being constructed.

GPUs are being purchased by the hundreds of thousands.

All of this infrastructure is largely optimized around one objective:

producing and deploying more artificial intelligence.

But if control becomes the next fundamental bottleneck, the architecture of those systems may eventually have to change.

Future AI infrastructure may require processors designed around enforceable permissions.

Hardware-level containment.

Cryptographic model identity.

Authenticated execution.

Isolated network environments.

Independent monitoring systems.

Hard limits on external access.

Physical shutdown mechanisms that software cannot override.

The data centers may still be built.

The electricity will still be required.

The networks will still matter.

But what sits inside those data centers may evolve.

And suddenly an entirely different question appears:

WHAT IS THE CHIP AFTER THE GPU?

That is not merely a safety question.

It is an investment question.

It is an Nvidia question.

It is a semiconductor question.

It is a data-center question.

It is a cybersecurity question.

And ultimately it may become a national-security question.

Because if artificial intelligence is already sufficiently capable to transform civilization, then perhaps humanity's most important engineering challenge is no longer creating more intelligence.

It is ensuring that intelligence remains under human authority.