When AI Labs Sound the Alarm, Ask for the Evidence

Key takeaways

• A warning about catastrophic AI harm does not establish how likely it is.
• Independent evaluations should examine both the claimed risks and the company’s response.
• Evaluators need access to underlying evidence and freedom to publish unfavorable findings.
• Future threats must not crowd out accountability for harms happening today.

When an AI lab says its technology could threaten humanity, it is making a claim about both danger and power. Developers such as OpenAI and Anthropic know their systems closely, but they also have products to sell. That makes independent scrutiny essential to deciding how much weight their warnings deserve.

Separate the scenario from the capability

Existential risk means something far beyond a chatbot giving a bad answer. It refers to threats that could destroy humanity or irreversibly damage its long-term future.

A claim that large needs to be unpacked. Describing a possible catastrophe is one step. Demonstrating that a system can cause it is another. Establishing whether the necessary conditions could realistically come together requires further evidence.

Suppose a model writes a dangerous plan. That establishes that it can produce the text. Can it access the systems needed to act? Acquire the necessary resources? Keep executing when a person tries to stop it?

Those questions determine how much the demonstration actually tells us.

The absence of a disaster does not establish safety, either. Evaluating risks before harm occurs is the point of prevention. But possibility, demonstrated capability, and likelihood need separate treatment. Bundle them together, and readers lose any useful sense of the risk.

Audit the warning and the response

AI developers hold information outsiders may struggle to obtain. They also face pressure to ship products and grow their businesses. That combination creates a reason for scrutiny without establishing dishonesty.

A dramatic warning can draw attention to a safety problem. It can also make the company sound extraordinarily capable. A warning label can do some of the work of a sales pitch.

Guessing which motive dominates gets us less far than examining the evidence. An independent investigation should ask three questions:

• Under what conditions did the behavior occur? A test resembling everyday use tells a different story from an experiment that grants a model unusually broad permissions.
• Can other evaluators reproduce the result? One striking demonstration needs follow-up to establish whether the behavior recurs.
• What changed after the company saw the result? Investigators should examine whether the developer restricted access, narrowed deployment, or took other concrete precautions.

That last question connects the company’s language to its decisions. If a developer describes a risk as severe, its response should reflect that assessment. The public needs evidence of what changed and why the company considers those changes adequate.

Independence requires access and publishing rights

An outside expert reading a company-selected packet of documents is still working inside boundaries the company controls. An impressive advisory board cannot resolve that problem on credentials alone.

Evaluators need access to evidence beyond the public report. That includes assessment methods, failure cases, and results from before and after safeguards were introduced. They should also disclose which requested materials they could not obtain.

They need publication rights, too. If the subject of an investigation can suppress an unfavorable conclusion, readers see only the findings it permits them to see. Funding arrangements and evaluators’ conflicts of interest should be disclosed.

None of this requires publishing every sensitive detail. Personal information needs protection. Some technical findings could enable misuse. A workable approach could give authorized independent evaluators access to sensitive evidence while making the assessment criteria and main conclusions public.

The investigation must also allow for either outcome. Strong evidence of danger should lead to stronger precautions. Weak evidence should lead to revised claims. An evaluation whose conclusion is fixed in advance offers little reason for public confidence.

Keep today’s harms in view

Threats to humanity’s future command attention. Fraud and privacy violations still require investigation, accountability, and remedies.

These problems call for different kinds of assessment. Claims about future catastrophe need explicit assumptions and a clear account of uncertainty. Harm that has already occurred requires evidence about its scale, its causes, and how affected people can obtain redress.

Collapsing everything into a single “AI risk” score makes it harder to identify the right response.

Policymakers need independent judgment as well. Developers’ expertise can inform rules, but adopting one company’s evaluation framework as the industry standard requires separate justification. Safety measures should be assessed for effectiveness, cost, and their effects on competition.

AI companies’ warnings deserve a serious hearing. Independent evaluators need the authority to inspect the evidence and examine what the company did about it. The more consequential the warning, the stronger the case for letting someone else check the work.