The Missing Steps Between AI Misbehavior and Catastrophe

Key takeaways

• Unexpected AI behavior does not, by itself, establish a path to human extinction.
• A test can reveal a dangerous capability without showing how often it will surface in everyday use.
• Evidence that supports taking precautions may fall short of evidence that supports predicting catastrophe.
• Practical AI safety depends on what a system can access, execute, and undo—and whether people can stop it.

An AI that drafts a terrible email gives you something to fix. An AI that sends it to your entire customer list gives you something to explain. That gap helps clarify the debate around Yann LeCun and Dario Amodei: what, exactly, is the danger being discussed?

Start with the claim, not the confidence

The claim that AI produces false information and the claim that AI could cause human extinction carry very different burdens of proof.

Skepticism about extinction risk does not necessarily mean dismissing fraud, privacy breaches, or other concrete harms. Equally, warning about a dangerous behavior does not necessarily mean predicting an imminent catastrophe.

When assessing the positions associated with LeCun and Amodei, start with the specific risk. Is the argument about flaws in systems people use today? Or is it about losing control of much more capable systems in the future?

The timeline matters, too. Whether an AI assistant is safe enough to use at work today is connected to the long-term question of controlling more powerful AI. But answering one does not settle the other.

“Optimist versus pessimist” makes a convenient headline. It tells you remarkably little about which claim the evidence supports.

“Runaway AI” leaves out the important details

Imagine a model in a test environment responding, “I will delete the administrator account.”

That is a hypothetical example, and the first finding would be narrow: the model produced those words. Did it issue a deletion command? Did it have permission to execute one? Was anything actually deleted?

If it executed the command, the finding becomes more serious. Even then, an action contained within a test environment and an action affecting external systems have different consequences.

To judge a dangerous result, ask:

• What instructions and conditions produced the behavior?
• Did the model describe an action or execute it through a tool?
• Did the behavior recur when the test was repeated?
• Could a person intervene and stop it?

These questions separate capability from frequency. A test can show that a system behaves dangerously under particular conditions. It cannot, by that result alone, tell us how often those conditions or behaviors will occur during ordinary use.

That is also no reason to wave the result away. If the same conditions can arise in a deployed service, the test may have exposed a weakness worth fixing before someone encounters it outside the lab.

Predicting disaster and preparing for it are different judgments

Two questions often get tangled together:

Do we have enough evidence to believe a catastrophe will happen? And do we have enough evidence to take precautions against it?

The first requires a convincing causal chain. Showing that an AI can deceive someone does not establish that it can seize control of a system. The argument must also explain how it would acquire the necessary resources and permissions, bypass safeguards, and sustain that control.

The second involves a judgment about action under uncertainty. If the potential damage is enormous, precautions may make sense before anyone can assign a precise probability. Still, being able to imagine a catastrophe does not make every scenario equally credible.

Someone can reasonably argue that extinction risk has not been established while also supporting stronger controls on AI systems. Those positions can coexist.

The proposed precaution needs scrutiny, too. Restricting a model’s access to a sensitive tool and restricting an entire field of technological development have different costs and consequences. Evidence sufficient to justify one does not automatically justify the other.

Permissions turn the argument into something practical

For anyone deploying AI at work, the immediate question is what happens when it makes a bad judgment.

Consider an email assistant. If it drafts a message for a person to review and send, there is an opportunity to catch a mistake. Give the same assistant access to a customer database and permission to send messages automatically, and that mistake can travel straight outside the company.

No agreement on extinction probabilities is required to examine that difference.

The useful checks are concrete: what can the system access, which actions can it execute, and which changes can be reversed? If it starts doing something harmful, can a person stop it before the damage spreads?

Requiring human confirmation for consequential actions creates a chance to intervene. Such controls help with ordinary mistakes as well as potentially deliberate harmful behavior. You do not have to resolve the question of machine intent to reduce the damage from a bad decision.

The LeCun–Amodei debate becomes more useful when we examine the evidence and conditions behind each claim. Whenever “runaway AI” appears, ask what the system actually did and how much control people retained. For the AI in your own workflow, start with its permissions.