The Machine That Lies Fluently: Aphyr's Unsettling Case Against ML

Kyle Kingsbury — better known as Aphyr, the person who built Jepsen and spent a decade proving that your favorite database is lying to you — has turned his attention to machine learning. His thesis is blunt: ML is, at its core, a technology for producing plausible falsehoods, and this property gets stronger, not weaker, as models improve. The title of his piece says it all: “ML promises to be profoundly weird.”

Why You Should Listen to This Particular Critic

If you’ve worked with distributed systems, you know Jepsen. It’s the testing framework that put MongoDB, Elasticsearch, CockroachDB, and dozens of other databases through hell — and published the results. Kingsbury’s entire career has been built on one question: does this system actually do what it claims?

That’s a consistency guy’s framing, and it’s exactly the lens he brings to ML. When he looks at a large language model, he doesn’t ask “how impressive is this?” He asks “can I trust this output?” His answer is not encouraging.

The Unfixable Bug

Traditional software has a useful property: when it breaks, you can trace the fault. A database returns the wrong value, you walk the code path, you find the bug, you patch it. The relationship between input and output is legible.

ML doesn’t work that way. When a large language model confidently produces wrong information, there’s no stack trace to follow. The error lives somewhere in billions of parameters where patterns combined in the wrong way — but pinpointing where, or why, is effectively impossible. Aphyr frames this as structural opacity. It’s not a bug you can fix. It’s a state where you can’t even determine whether something is a bug.

Here’s the part that should make you uncomfortable: as models get larger and more capable, the lies get better too. GPT-2’s hallucinations were obvious. You could spot them in seconds. The latest generation of models can fabricate information convincing enough to fool domain experts. The failure mode isn’t getting worse — it’s getting harder to detect.

What “Profoundly Weird” Actually Means

Aphyr chose that phrase deliberately. In the entire history of human information systems, nothing has worked quite like this.

Books have authors. News goes through editorial review. Wikipedia demands citations. Over thousands of years, human societies built an elaborate infrastructure for answering one question: should I believe this? Author credentials, institutional verification, peer review, source tracking — all of it exists to establish trust.

ML sidesteps the entire apparatus. Its output has no author. There’s no editorial process. Ask it for sources and it will invent papers that don’t exist. The problem isn’t that the output is wrong — it’s that the output is wrong while being perfectly formatted, syntactically flawless, and deeply persuasive. Form without truth. That’s the “profoundly weird” part.

An Epistemological Crisis at Scale

This isn’t a technical limitation waiting for a better RLHF technique. It’s an epistemological crisis — a fundamental challenge to how we determine what’s true.

The evidence is already piling up. Fake citations generated by AI have been found embedded in published academic papers. Lawyers have filed briefs citing hallucinated case law. AI-generated code confidently implements wrong logic in ways that pass casual code review. These aren’t hypothetical risks. They’re Tuesday.

What makes Aphyr’s framing particularly sharp is his emphasis on scale. A human liar has limited bandwidth. An ML model can generate thousands of plausible falsehoods per second. Those falsehoods accumulate on the internet. The next generation of models trains on that data. You get a self-reinforcing cycle of epistemic contamination — fabrication feeding fabrication, compounding with each iteration.

The Cost of Truth Is Going Up

Aphyr doesn’t offer a tidy solution, and that’s honest of him. He’s a diagnostician, not a salesman. Just as Jepsen’s value was in precisely identifying where systems failed their promises, his contribution here is naming the problem clearly.

A few directions emerge from his argument. Default to distrust with ML output. Treat any claim not linked to a verifiable source as provisional. And — most fundamentally — start building the new epistemological tools and social infrastructure that the ML era demands. Individual critical thinking isn’t enough when search engines are polluted with generated content and academic databases are diluted with synthetic papers. This is a systems problem, not a literacy problem.

Aphyr’s essay ultimately asks a question worth sitting with: are we entering an era where the cost of truth is rising sharply while the cost of fabrication approaches zero? A world where lies are infinite and free, but verification demands ever more effort. How many times today did you accept AI-generated text without checking?