AI Can Write the Code. Who Owns the Outage?

Key takeaways

• Margaret Hamilton led the Apollo flight software team at MIT’s Instrumentation Laboratory.
• AI-written code needs verification for failures, interruptions, and repeated requests.
• Passing tests cannot establish whether the requirements behind those tests are correct.
• Teams should assign deployment approval and incident response responsibilities before code reaches production.

An AI-written checkout flow can look finished right up until a dropped connection creates two orders. Margaret Hamilton’s work on Apollo flight software offers a useful way to think about what a convincing demo leaves out: how a system behaves when things go wrong.

Apollo’s lesson starts with failure

Hamilton led the Apollo flight software development team at MIT’s Instrumentation Laboratory. The software was part of the machinery that made the mission possible. Its failures could have consequences well beyond an error message.

In that environment, implementing the requested features was only part of the job. A system also needed rules for unexpected inputs and competing demands. Which tasks should keep running? Which could wait? When should a person be alerted?

The lesson to carry into AI-assisted development is designing for failure. Expected behavior and failure behavior belong in the same conversation.

An online store does not carry the same stakes as a spacecraft. It still needs an answer to the question: what happens when our assumptions stop holding?

One click, two orders

Suppose you ask an AI coding tool to build an ordering feature. A customer selects a product, clicks the button, and gets an order confirmation. The demo looks good.

Now interrupt the connection immediately after the server saves the order. The customer sees an error, assumes nothing happened, and clicks again. The store may now have two orders for one intended purchase.

This is where idempotency matters: repeating the same request should not duplicate its intended effect. Giving an order request a unique identifier lets the server recognize a previously processed request and avoid creating another order.

An AI tool can help implement that behavior. But the team still has decisions to make:

• What identifies a retry of the same order?
• When should the customer be allowed to retry?
• What should the interface show when the outcome is uncertain?

Those decisions define the feature just as much as the button and database write do. If the request to the coding tool leaves them out, the implementation may leave them out too.

The gap between a working demo and a dependable service often lives in these details.

Green tests can confirm the wrong thing

“If the tests pass, does it matter who wrote the code?”

Tests are essential. They check whether specified conditions produce expected results. They do not establish that the expected results are correct.

Consider a coupon offering a discount above 100%. Before testing the arithmetic, someone has to decide whether the store should accept that coupon at all. A calculation can be mathematically correct while violating the service’s rules.

Give an AI tool the same mistaken premise for both implementation and testing, and both outputs can agree perfectly. The test suite turns green. The mistake survives.

Requirements review therefore needs to cover questions such as:

• What happens when an input falls outside the allowed range?
• How long should the system wait for an external service?
• If a task stops halfway through, what counts as completed?
• What result should a repeated request return?

AI can suggest cases and help turn them into tests. The team still needs to judge whether those cases reflect the actual service rules and risks.

Agreement between code and tests is useful evidence. Agreement with the requirements needs its own review.

Assign responsibility before deployment

“The AI wrote it” is not much of an incident response plan.

When something breaks, someone needs authority to stop further changes, assess the damage, and decide how to recover. Those responsibilities should be explicit before the code reaches production.

That means naming who reviews requirements, who approves deployment, and who responds first to an incident. In a small team, one person may fill several roles. The responsibilities still need to be clear.

Permissions should reflect those responsibilities. A tool that proposes code has a different reach from one that can directly modify production data. Broader access calls for tighter limits, a record of changes, and a workable recovery path.

Review effort should also match the consequences. Changing a button label and deleting customer records do not need identical procedures. Changes that are difficult to reverse—or costly to get wrong—deserve more scrutiny.

Hamilton’s work is a reminder that building software includes anticipating its failures. As AI makes code easier to produce, teams still have to define correct behavior, verify it, and assign responsibility for what happens next. Before deployment, know who can stop the change—and how they will recover if it fails.