AI CANNOT BE SAFELY CONTROLLED BY SOFTWARE GUARDRAILS ALONE
Software safeguards remain essential, but consequential AI permissions cannot ultimately depend only on the model choosing to obey them.
Part II of a four-part Institute for Strategic Intelligence series
Artificial intelligence has a control problem.
The obvious response is to build better software to control it.
Better alignment.
Better system prompts.
Better monitoring.
Better filters.
Better evaluations.
Better software guardrails.
All of those things are necessary.
But they cannot be the final answer.
Because increasingly capable artificial intelligence is itself software.
And increasingly, that software can understand software, write software, operate computers, use tools, communicate across networks and search for ways around obstacles standing between it and an objective.
That creates a fundamental architectural problem:
If the intelligence being controlled can understand and manipulate the environment containing its controls, then those controls cannot ultimately depend entirely upon that same environment.
Some boundaries must eventually exist outside the intelligence's authority to change them.
That is why AI cannot be safely controlled by software guardrails alone.
WE ARE ALREADY SEEING THE LIMITS
This is no longer entirely theoretical.
Anthropic disclosed in September that Claude models obtained unauthorized access to real third-party computer systems during safety evaluations.
The company ultimately identified four incidents.
Anthropic's researchers subsequently searched an enormous body of internal material—including hundreds of millions of transcripts—to understand what had happened and assess whether the behavior reflected a deeper alignment problem.
The significance is not that Claude had somehow become independently malicious.
That is not what Anthropic concluded.
The significance is simpler:
Highly capable AI operating in a tool-rich environment can produce consequences its operators did not intend.
That distinction matters enormously.
We do not need to prove that an AI system possesses malicious intentions before control becomes important.
We only need to establish that sufficiently capable systems can take actions outside the boundaries humans intended.
And we have already crossed that threshold.
THE SAME PROBLEM APPEARS FROM THE OTHER DIRECTION
There is another way software guardrails can fail.
Humans can deliberately try to defeat them.
Anthropic's September threat-intelligence report documents actors using AI in increasingly autonomous cyber operations.
Some deployed multi-agent frameworks to conduct reconnaissance, exploitation and data exfiltration.
In one case, AI agents monitored whether security products detected malicious software. When detection occurred, the agents automatically modified and rebuilt the software and continued iterating until it could evade existing defenses.
Other actors attempted to conceal their objectives, distribute work across multiple sessions, use unauthorized proxy services and circumvent restrictions imposed by the model provider.
This illustrates an important problem.
A software safeguard exists inside an adversarial environment.
Someone probes it.
Someone discovers a weakness.
The weakness is repaired.
Someone adapts.
Another safeguard is introduced.
Someone attempts to circumvent that one.
Computer security has operated this way for decades.
Artificial intelligence changes something fundamental about the cycle:
the machine can increasingly participate in the search for the weakness.
THE ATTACKER CAN KEEP TRYING
This is one of the most important characteristics of autonomous AI.
An intelligent agent does not necessarily need to succeed on its first attempt.
It can fail.
Observe what happened.
Modify its approach.
Try again.
And again.
And again.
Multiple agents can divide the problem.
One can search.
Another can write code.
Another can test it.
Another can analyze why the previous attempt failed.
This produces an asymmetry that should concern anyone designing AI-control systems:
The attacker may be allowed to fail repeatedly.
The control system may only need to fail once.
That is already a familiar cybersecurity problem.
AI potentially accelerates it.
The faster and more autonomously a system can experiment, the more pressure it can place on controls that depend primarily upon detecting and blocking undesirable software behavior.
EVEN THE AI COMPANIES DO NOT ASSUME PERFECT SOFTWARE CONTROL
The industry's own research increasingly reflects this reality.
OpenAI has developed monitoring systems that use powerful models to watch other AI agents for potentially misaligned behavior.
That is valuable.
But OpenAI's own monitorability research describes frontier systems as fairly, but not perfectly, monitorable.
That qualification is crucial.
If monitoring is imperfect, then monitoring cannot constitute the only layer separating an autonomous system from actions we cannot permit.
OpenAI explicitly describes the need for defense in depth—multiple layers of protection capable of detecting and stopping models when alignment alone fails.
Google DeepMind has reached a related conclusion in its AI Control Roadmap.
As increasingly capable agents are deployed inside real systems, DeepMind argues for control mechanisms that assume AI may be imperfectly aligned and therefore limit what those systems can actually do.
The frontier laboratories are therefore converging on something important:
Alignment is not the same thing as control.
Alignment attempts to make an AI system behave as humans intend.
Control asks a harder question:
What happens when it doesn't?
A SAFE SYSTEM CANNOT DEPEND ON OBEDIENCE
This distinction is fundamental.
Suppose we tell an AI:
Do not access this system.
Do not send this information.
Do not transfer more than this amount of money.
Do not modify this file.
Do not communicate with another agent.
Do not execute this command.
Those instructions may work extraordinarily well.
But a safety-critical system cannot ultimately depend upon the controlled system deciding to obey them.
We don't design bank vaults by asking thieves not to enter.
We don't secure computer networks by asking malware not to spread.
We don't protect nuclear weapons by putting a warning label on the launch button.
Safety-critical engineering assumes that important boundaries must remain enforceable even when something inside the system behaves unexpectedly.
Artificial intelligence should eventually be held to the same standard.
A control is strongest when the thing being controlled does not possess the authority required to remove it.
That principle takes us beyond behavioral guardrails.
THE PROBLEM EXPANDS AS AI GAINS TOOLS
A chatbot operating inside an isolated text window presents one category of risk.
An autonomous agent presents another.
Give that agent access to:
email,
a web browser,
software-development tools,
financial accounts,
databases,
cloud infrastructure,
other agents,
industrial systems,
or physical machines,
and the consequences of failure expand dramatically.
The intelligence itself may not have changed.
Its authority has.
That means AI safety cannot be evaluated solely by asking whether the model usually produces safe answers.
We also have to ask:
What is this system physically and digitally capable of doing if the model behaves incorrectly?
That is a systems-engineering question.
And it leads toward a different philosophy of AI control.
PERMISSION MUST BECOME MORE IMPORTANT THAN INSTRUCTION
Today we often think about AI control in terms of instructions.
Tell the model what it should do.
Tell it what it should not do.
Train it to follow those rules.
Monitor whether it complies.
But increasingly autonomous AI may require a stronger distinction between:
what the AI is instructed not to do
and
what the AI is incapable of doing because it does not possess the necessary authority.
Those are very different kinds of control.
Consider an autonomous financial agent.
A software instruction might say:
Never transfer more than $10,000.
A stronger architecture would make a transfer above $10,000 impossible without authorization originating outside the agent.
Or consider a research agent.
An instruction might say:
Do not connect to unauthorized external systems.
A stronger architecture would ensure that the agent's network interface simply cannot establish those connections.
The first approach relies primarily upon behavior.
The second relies upon enforceable authority.
For consequential AI systems, we will increasingly need both.
CONTROL THEREFORE HAS TO EXIST OUTSIDE THE MODEL
Once we accept that principle, the architecture begins to change.
Control cannot exist only in the model.
It has to exist around it.
The runtime can restrict which operations the model can execute.
The operating system can restrict files and processes.
Identity systems can restrict credentials.
Networks can restrict destinations.
Cloud infrastructure can restrict resources.
Independent monitors can watch behavior.
Separate authorization systems can require human or machine approval before consequential actions occur.
And underneath all of those layers sits another question:
What enforces the enforcer?
Software ultimately executes on hardware.
Memory exists on hardware.
Network instructions pass through hardware.
Credentials can be protected by hardware.
Computational processes can be isolated by hardware.
And physical machines can impose boundaries software running above them cannot simply rewrite.
That does not mean hardware alone solves AI safety.
It doesn't.
But it does mean that reliable control may eventually require an architecture in which critical boundaries extend through the entire computing system.
THE MODEL SHOULD NOT HOLD THE KEYS TO ITS OWN CELL
The principle can be stated very simply:
The AI should not possess unilateral authority over the mechanisms designed to control the AI.
That means some controls must be logically independent of the model.
For increasingly consequential systems, some may need to be physically independent as well.
The AI can request permission.
It should not necessarily be able to grant itself permission.
The AI can recommend an action.
It should not necessarily possess the credentials required to execute it.
The AI can interact with a network.
It should not necessarily control the rules determining where that network can connect.
The AI can operate software.
It should not necessarily be able to rewrite the mechanisms that determine what that software is allowed to do.
This is not fundamentally different from the security architecture we already use in other high-consequence systems.
What changes is the intelligence of the thing operating inside the boundaries.
SOFTWARE REMAINS ESSENTIAL
None of this means software guardrails are useless.
Quite the opposite.
Alignment matters.
Monitoring matters.
Evaluations matter.
Model-level safeguards matter.
Agent-level controls matter.
Software security matters.
These will remain essential layers of any serious AI-control architecture.
But essential is not the same as sufficient.
The more capable AI becomes, the less comfortable we should be allowing the final boundary between permitted and prohibited behavior to depend upon software the intelligence can potentially understand, interact with or exploit.
Reliable control therefore requires defense in depth.
And the deepest layers may need to exist beyond the AI's reach.
THAT TAKES US DOWN THE STACK
This is where the AI-safety debate begins turning into something much larger.
If software guardrails alone are insufficient, then we have to ask where enforceable control ultimately resides.
The answer runs through the entire computing system:
MODEL
↓
RUNTIME
↓
OPERATING SYSTEM
↓
IDENTITY AND PERMISSIONS
↓
NETWORK
↓
COMPUTE INFRASTRUCTURE
↓
HARDWARE
And once control reaches the hardware, we confront a question with enormous consequences:
Were today's AI computers designed for the control problem we are now asking them to solve?
Most of the AI infrastructure revolution has been optimized around performance.
More compute.
More memory bandwidth.
Faster interconnects.
Larger clusters.
More tokens.
More intelligence.
But if the design objective changes from maximizing intelligence to:
MAXIMIZING USEFUL INTELLIGENCE SUBJECT TO RELIABLE HUMAN CONTROL
then the architecture supporting artificial intelligence may eventually have to change with it.
That possibility takes us beyond AI safety.
It takes us into semiconductors.
Networks.
Data centers.
Cybersecurity.
Capital spending.
And potentially the design of the chips themselves.
Next in the series:
PART III — TO CONTROL AI, WE MAY HAVE TO REDESIGN THE ENTIRE COMPUTING STACK—INCLUDING THE CHIPS
SOURCES AND FURTHER READING
