SQLite Just Lost a 16-Year Staring Contest

SQLite is the software everyone points to when they want to say testing works. Hundreds of lines of test code per line of shipped code. Running on airplanes, in browsers, on every phone in your pocket. Then a data corruption bug that had been sitting there for 16 years surfaced — after it had already quietly mangled real data at Tailscale.

The Bug That Doesn’t Announce Itself

It started the way these things usually do: an internal report that something looked off. Data Tailscale kept in SQLite had stopped being internally consistent. No crash. No error in the logs. The database reported itself perfectly healthy. The contents just weren’t what they should have been.

This is the worst category of bug. A crash at least leaves a body and a timestamp. A system that silently hands back wrong answers gives you nothing — you can’t tell whether it started last Tuesday or last quarter. You’re not debugging a failure, you’re doing archaeology.

WAL, and the Trap Inside the Reset

The fault lived in SQLite’s WAL mode — Write-Ahead Logging. The idea is simple: don’t edit the manuscript, keep a running notepad of changes.

The main database file stays untouched while writes pile up in a separate WAL file. Readers merge the two to see current state. Writers only append to the notepad. That’s why readers and writers stop blocking each other, and why WAL mode outruns the default rollback journal for most workloads.

Eventually the notepad gets long and needs folding back in. That’s a checkpoint: apply the accumulated changes to the main file, then start writing the WAL from the top again. That “from the top again” step is the WAL reset.

The defect was a boundary condition in exactly that moment. Under the right timing, a reset could make parts of an already-committed transaction cease to exist. SQLite saw nothing wrong. Data that had received a commit acknowledgment simply evaporated.

Sixteen Years Is a Long Time to Hide

So how does a bug survive the most tested codebase in the industry for a decade and a half?

One reason: you can’t reproduce it on purpose. This class of failure needs writes to land in a specific order, a checkpoint to trigger at a specific size, and the process to die at exactly that instant. Three conditions, all lining up. In production, the odds are somewhere between rare and lottery-ticket.

Conventional tests assume the sane ordering. And 100% code coverage buys you nothing here, because coverage measures whether a line ran — not what order and timing it ran in relative to everything else. The number of distinct interleavings is, practically speaking, infinite. Coverage is a map of the roads; concurrency bugs live in the traffic.

There’s a second reason, and it’s less flattering to all of us. The damage is invisible in most deployments. If a handful of rows drift in your average SQLite-backed app, nobody notices. The app doesn’t crash, so no bug report gets filed. It took a place like Tailscale — where data consistency is the product — for the symptom to become legible at all.

The Tool That Finally Caught It

The bug was pinned down by Antithesis, a deterministic simulation testing platform.

The premise: run the software inside a fully controlled virtual environment where the platform owns everything nondeterministic. The clock. Thread scheduling. Disk I/O ordering. Network latency. Then be hostile inside that world — kill processes at arbitrary moments, truncate writes mid-flight, shuffle thread execution into orders the real world would only produce once in a billion runs.

The word doing the heavy lifting is deterministic. Feed it the same seed and you get the identical execution, byte for byte. The crash you saw once, you can have again, on demand, forever. You can rewind from the moment of failure and walk backward toward the cause.

Compare that to the normal experience. You grep logs. You guess at repro conditions. You wait, hoping it happens again while you happen to be watching. That’s a multi-week effort with a real chance of never converging. Deterministic simulation changes the job from finding a bug to replaying one you already have.

None of this is new, exactly. FoundationDB has been validating distributed database correctness this way for over a decade, and Antithesis was founded by people who came out of that team. TigerBeetle has built its whole reputation on a similar philosophy. What’s changed is that the approach is escaping the small circle of teams willing to build their own simulator from scratch.

What to Actually Take From This

A few things stuck with me.

Treat “battle-tested” as a probability, not a guarantee. SQLite remains excellent software — taking 16 years to surface is itself evidence of how solid it is. But zero-bug software doesn’t exist. The realistic posture is to assume something in your stack has a mine nobody has stepped on yet, and design as though that’s true.

Silent corruption is more dangerous than a crash. Crashes page you. Corruption waits. Which means the burden falls on you to go looking: run integrity checks on a schedule, and encode the invariants your data must satisfy as executable assertions, not tribal knowledge. One nightly job checking “this counter should never exceed that one” can catch months of drift you’d otherwise find during an incident.

Coverage and reliability are different axes. Coverage tells you a line executed. It says nothing about concurrency, partial writes, or crash consistency. If you’re building anything stateful, you need tests that deliberately manufacture weird orderings — fault injection, chaos under controlled conditions, or full simulation if the stakes justify it.

The Part That Lingers

The most tested code on the planet was hiding a 16-year-old bug. The thing that found it wasn’t another thousand unit tests. It was simulating an entire universe and forcing the worst possible sequence of events to happen on purpose.

Whatever database, queue, or cache you’re running right now probably has something similar buried in it. The interesting question isn’t when it goes off — it’s whether you’d notice. If your data silently corrupted itself tonight, what in your system would tell you? Start there.