WHAT IF THE PEOPLE BUILDING AI BELIEVE IT MAY BE DANGEROUS—AND BUILD IT ANYWAY?
Jacob Coxon’s resignation exposes the deeper AI-safety problem: even companies that sincerely fear loss of control may believe competition makes slowing down impossible.
Jacob Coxon spent three years helping build frontier artificial-intelligence systems at OpenAI and Anthropic. On September 8, he resigned from Anthropic—and said he was leaving the AI industry entirely.
His warning was blunt: the leading laboratories are racing toward self-improving superintelligence and, in his judgment, “gambling with our lives.”
It would be easy to treat this as another dramatic prediction about artificial intelligence. That would miss the most important part of the story.
Coxon is not an outside critic speculating about technology he has never seen. He worked inside two of the organizations closest to the frontier. Nor does he describe Anthropic’s safety program as fraudulent. According to The Wall Street Journal, he believes the company’s safety work is sincere.
That distinction makes his warning more consequential.
The problem he identifies is not simply that irresponsible people may build dangerous systems. It is that competitive pressure may make responsible behavior impossible—even for a company genuinely trying to behave responsibly.
THE DISTURBING QUESTION
What happens when the people building the most powerful technology in history believe it could become uncontrollable, but also believe they cannot afford to stop building it?
That is no longer merely a technical alignment problem. It is a coordination problem, a governance problem and a competition problem.
A frontier laboratory can believe that slowing down is prudent and still conclude that slowing down alone would be reckless. If it pauses while a competitor continues, the competitor may gain the decisive capability first. If every laboratory thinks this way, individually rational decisions can produce a collectively irrational outcome.
Each company can say:
- We understand the danger.
- We are investing heavily in safety.
- We would prefer the industry to move more carefully.
- But we cannot unilaterally fall behind.
The result is a race that no participant may fully endorse and no participant believes it can safely leave.
This is the deeper meaning of Coxon’s resignation.
SINCERITY MAY NOT BE ENOUGH
The comforting version of the AI-safety debate divides the world into responsible companies and irresponsible companies. Under that model, the solution is to support the responsible firms, demand better internal safeguards and punish reckless behavior.
But Coxon’s account points toward a harder possibility: sincere safety work may coexist with a system of incentives that continually weakens its practical effect.
A company may establish safety teams, publish risk frameworks, test new models and warn publicly about catastrophic danger. Yet when a capability threshold approaches, it still faces pressure from investors, customers, governments and competing laboratories. The question becomes not whether the company values safety, but whether it will accept losing strategic position in order to preserve it.
There is little evidence that any major laboratory, nation or capital market is prepared to make that sacrifice unilaterally.
This does not prove that artificial intelligence will escape human control. Coxon’s catastrophe scenario is a forecast, not an established fact. Experts disagree sharply about the probability, timing and even technical plausibility of self-improving superintelligence.
But uncertainty does not eliminate the problem. It defines it.
When the potential consequence is catastrophic, “we do not know” cannot be treated as equivalent to “it is safe.”
THE WARNING IS PART OF A LARGER PATTERN
Coxon’s departure should not be evaluated in isolation. It belongs beside a widening body of evidence:
- Researchers and safety leaders leaving frontier laboratories.
- Internal warnings that future systems may be difficult to align or control.
- Models discovering unexpected strategies during testing.
- Guardrails that can be weakened, bypassed or removed.
- Autonomous agents taking actions their operators did not specifically direct.
- AI systems being used to scale cyber reconnaissance and attack.
- Laboratories modifying safety policies as competitive conditions change.
- Governments struggling to regulate systems whose capabilities advance faster than legislation.
No single incident proves loss of control. Together, however, they reveal a recurring gap between the speed at which AI capability is advancing and the institutions available to govern it.
Cybersecurity makes that gap especially visible. An AI system does not need to become a science-fiction superintelligence to create serious danger. It only needs enough autonomy, persistence and adaptability to probe defenses faster than humans can respond.
The same underlying issue appears in both domains: systems are gaining the ability to act, learn and iterate at machine speed while oversight remains organized at human speed.
THE RACE CHANGES THE STANDARD OF EVIDENCE
Ordinarily, society waits for clear evidence of harm before imposing costly restrictions. That approach becomes dangerous when the first conclusive evidence could arrive only after control has been lost.
A bridge can be load-tested before millions of people cross it. A drug can be evaluated before mass distribution. But a highly capable, self-improving system may not permit the same sequence. If deployment itself creates the conditions for rapid improvement, replication or strategic autonomy, the experiment and the consequence may become inseparable.
This is why insider testimony matters even when it is not definitive proof.
The people closest to frontier development possess incomplete information, conflicting incentives and different beliefs. Their statements should not be accepted uncritically. But departures, warnings and policy disputes are evidence about how the people building these systems perceive the risk.
They belong in the historical record.
WHAT RESPONSIBLE GOVERNANCE WOULD REQUIRE
If competitive pressure is the mechanism driving risk, better intentions inside individual companies will not be sufficient. Governance must change the incentives facing all major participants.
That could require:
- Common capability thresholds. Laboratories should not decide independently when a model is dangerous enough to require additional controls.
- Independent evaluation. Safety claims should be tested outside the company that developed the model.
- Mandatory incident reporting. Unexpected autonomous behavior, containment failures and serious guardrail breaches should enter a shared evidentiary record.
- Limits tied to demonstrated risk. The most consequential training runs and deployments may require enforceable conditions rather than voluntary promises.
- International coordination. A national pause that simply transfers the race elsewhere does not solve the underlying problem.
None of these steps is simple. Some may slow beneficial innovation. Some may be difficult to enforce. Governments may misuse safety arguments to protect incumbents or restrict open research.
But those are arguments for designing governance carefully—not for pretending the coordination problem does not exist.
WHY THE INSTITUTE IS CREATING A NEW TRACKER
The Institute for Strategic Intelligence is establishing an AI — Safety & Cyber Security Tracker because the relevant evidence is now appearing across categories that are too often treated separately.
Loss-of-control concerns, autonomous-agent behavior, alignment failures, guardrail removal, AI-enabled cyberattack, insider warnings, safety-policy changes and regulatory responses increasingly form one story.
The tracker will maintain a dated, sourced record across two connected domains:
AI Safety & Control — evidence concerning alignment, autonomy, deception, containment, self-improvement, internal warnings, departures and governance.
AI Cyber Security — evidence concerning AI-enabled attacks, autonomous cyber operations, vulnerability discovery, defensive failures, agent swarms and institutional responses.
The objective is not to assume catastrophe. It is to distinguish verified events from forecasts, measure the accumulation of evidence and make changes in the risk environment visible over time.
THE QUESTION COXON LEAVES BEHIND
Coxon may be wrong about the outcome. He may be wrong about the timeline. The industry may develop technical safeguards strong enough to keep increasingly capable systems under meaningful human control.
But his central warning cannot be dismissed merely because it is frightening.
If even sincere companies believe they must continue a race they consider potentially catastrophic, then the safety of artificial intelligence cannot depend on corporate sincerity alone.
The question is no longer simply whether the people building AI understand the danger.
It is whether understanding the danger gives them enough power to stop.