AI Is Answering 911 Calls Now. Humans Wave Through a Third of Real Threats.

Every AI agent product ships with the same safety promise: before it does anything dangerous, it asks you first. A dialog pops up. You read it. You approve or deny. The human stays in control.

Nobody bothered to check whether that actually works. When someone finally did, the answer was ugly.

People approved one in three threats

The standard permission model is simple. An agent wants to delete a file, move money, or send data somewhere external, so it surfaces a confirmation prompt. A person reviews and clicks. This is human-in-the-loop, and the entire safety argument rests on it: a human holds final authority, therefore the system is safe.

An experiment put that assumption on the table. Researchers built a game-like environment and ran roughly 40,000 trials, asking people to approve or reject commands requested by an AI agent. Roughly one in three clearly dangerous commands sailed straight through.

These weren’t careless people. For the first several prompts, humans do fine. The failure shows up around prompt twenty, prompt fifty, when the brain quietly reclassifies the dialog from “information to read” to “obstacle to clear.” That’s permission fatigue. It’s the same reflex that makes you tap through an app’s permission screen without reading a word, and it’s not a character flaw — it’s how attention works under repetition.

A liability shield, not a safety mechanism

Here’s the uncomfortable reading. Is the approval button there to prevent accidents, or to relocate blame after one?

Building an approval UI is easy. Actually detecting which operations are dangerous is hard. So the hard problem gets outsourced to the human, and the system writes a log line: user approved. When something goes wrong, that log becomes a shield. We hand final judgment to someone who misses a third of the threats, then hold them responsible for the misses.

The two clicks aren’t even weighted equally. Deny, and work stops — you have to re-explain, dig into why the agent asked for that in the first place, restart a chain of reasoning. Approve, and work continues. The system is tilted toward yes, and people flow downhill.

Now put that button on a dispatch console

Several US jurisdictions now have AI answering 911 calls before a human does. The reasoning is straightforward and not stupid: dispatch centers are chronically understaffed, phones ring nonstop, and people die on hold. Using AI to triage non-emergency calls so human dispatchers can concentrate on real emergencies is a defensible idea on paper.

The question is what that one-in-three number means in this room.

When a coding agent’s approval slips through, a file gets deleted. You restore from backup. When a 911 triage call goes wrong, someone dies, and there is no restore. Worse, dispatchers may be the most permission-fatigued workforce on earth. They take hundreds of calls a shift, most of them not emergencies, and now their screen shows an AI-generated summary and a classification. Asking them to interrogate that summary from scratch, every time, for eight hours, is asking for something humans cannot do.

There’s a second problem. A 911 call is close to the worst-case input for a speech model. The caller is crying. Sentences break mid-word. There’s shouting, background noise, and English is frequently not their first language. A model that performs beautifully on clean text transcripts has not earned any presumption that it performs on that.

The honest counterargument

Fairness demands the other side. Nobody should block AI here on the theory that the current system is fine. It isn’t. Humans miss things today. People die on hold today. The right comparison isn’t AI versus a perfect human system — it’s AI versus the flawed human system actually running right now.

And there’s real work AI does well here. Pinpointing a caller’s location instantly. Live-transcribing the call. Surfacing similar past incidents on the dispatcher’s screen. Detecting the audio signature of possible cardiac arrest and flagging it. All of that augments judgment instead of replacing it. That’s the line that matters.

The failure mode is when the rationale drifts. Deployment starts as let’s help our dispatchers and quietly becomes maybe we need fewer dispatchers. In any budget-pressured agency, that drift is close to automatic.

What to actually change

The lesson from those 40,000 runs isn’t stop putting humans in the loop. It’s stop claiming a system is safe because there’s a human in the loop. A few things follow.

Cut the number of prompts. There’s a hard ceiling on how many judgment calls a person can make in a day. A system that asks 500 times isn’t getting 500 reviews — it’s logging 500 non-reviews. Auto-approve low-risk actions and reserve the interrupt for the genuinely dangerous minority.

Separate reversible from irreversible. If an action can be undone, just run it and offer an undo button. If it can’t be undone, a single click is not enough friction. 911 is squarely in the second category.

Measure approval rates. If a given operator approves 99 percent of prompts, that person isn’t reviewing — they’re clicking. This number sits right there in the logs, and almost nobody looks at it.

Thin material

Missing a third of threats across 40,000 runs means human attention is too thin a material to serve as the safety mechanism for automation. Using that material in a coding tool is one thing. Using it on a dispatch console is another argument entirely, and it deserves to be made explicitly rather than inherited by default.

Picture the last approval dialog your AI tools showed you. When did you last read one all the way through before clicking? If nothing comes to mind, that’s the real tensile strength of the safeguard you’re relying on.