Your LLM Is a Zip File. That's Not the Insult You Think It Is

In 2023, Ted Chiang wrote in The New Yorker that ChatGPT is a blurry JPEG of the web. Three years later it’s still the single most-quoted line in AI skepticism. The strange part is that people on the exact opposite side say the same thing — compression is intelligence — and mean it as praise. One fact, two religions. Worth untangling.

Prediction and compression are the same thing

Start with the premise, because it isn’t a metaphor. Compression and prediction are mathematically equivalent.

The logic is simple. To store data compactly, give short codes to frequent patterns and long codes to rare ones. But knowing what’s frequent is the same as knowing what comes next. Better prediction, better compression. Always.

Arithmetic coding makes this embarrassingly literal. If your model assigns probability p to the next symbol, the cost of encoding that symbol is exactly -log2(p) bits. Guess 0.5 and you pay 1 bit. Guess 0.25 and you pay 2. Guess 0.99 and you pay about 0.014 bits. Your confidence converts directly into file size.

Here’s where it gets interesting. That is precisely what an LLM does — emit a probability distribution over the next token. Nothing more. Which means you can drop an LLM into the probability-model slot of an arithmetic coder unmodified. No adapter, no retraining. It already speaks the right language.

Run it yourself and gzip looks pathetic

This experiment resurfaces on Hacker News every few months, mostly because the code is short. Under 100 lines of Python gets you a working compressor.

The numbers are lopsided. Gzip shrinks English text to roughly a third. Modern heavy-duty algorithms bottom out near a quarter. Point a real language model at the same text and you get under a tenth. DeepMind’s 2023 paper compressed text to 8.3% of the original using Chinchilla 70B. The stranger result in that same paper was outside text entirely: a model trained only on text compressed images better than PNG — 43.4% versus 58.5% — and beat FLAC on audio.

The gap makes sense once you look at what each system knows. Gzip’s entire worldview is “I saw that string earlier, here’s a pointer to it.” A language model is reasoning along the lines of: this is contract boilerplate, the parties were already defined in the preamble, so the next token is overwhelmingly likely to be the second party’s name. Grammar, structure, common sense, and general knowledge about how the world works all cash out as saved bits.

The case for compression as intelligence

This is where the compressionists plant their flag, and their standard-bearer is DeepMind’s Marcus Hutter. Since 2006 he’s run the Hutter Prize, a cash competition to compress 1GB of Wikipedia, with the payout scaled to how much you beat the standing record by. His argument is blunt: compressing Wikipedia better requires understanding Wikipedia better, so a compression contest is an intelligence contest.

It’s a defensible position. Squeeze text to its theoretical floor and you eventually have no choice but to model what the sentences mean. The theory underneath is serious too — Solomonoff induction, Kolmogorov complexity, the whole shortest-description-is-the-best-theory tradition. Occam’s razor, rewritten in math.

Ilya Sutskever has made the same argument repeatedly. To predict the next word well, you have to model the world that produced it. His example: to correctly predict the final sentence of a detective novel, the one naming the killer, you need to have followed the entire plot. Surface statistics won’t get you there.

Where the other side pokes holes

The rebuttals are strong. Three come up constantly.

First, you forgot the model. Yes, 1GB became 80MB. But the 70B model doing the work is 140GB, and you can’t decompress without shipping it too. Count everything and gzip wins by a mile. The DeepMind authors know this and argue the model cost amortizes across many files — fair enough. But by the traditional rules of the compression game, where you score decoder-inclusive size, the scoreboard reads very differently.

Second, compression is necessary, not sufficient. Intelligence implies good compression. Good compression does not imply intelligence. A pocket calculator beats you at multiplication and nobody calls it intelligent. Collapsing a high-dimensional concept into one scalar is a category error, and it’s the same error that makes benchmark leaderboards so easy to game.

Third, the lossy and lossless arguments keep getting mixed up. Chiang’s blurry JPEG is a claim about lossy compression: the original can’t be recovered, so the model confabulates something plausible. That’s the hallucination critique. The arithmetic-coding demo is lossless — bit-exact reconstruction, every time. These are different claims about different things, and debates slide between them constantly without anyone noticing.

Yes it’s a compressor. So what.

The framing is what breaks this argument. Asking whether an LLM is a compressor or an intelligence produces no answer, because the first part isn’t in dispute. It is a compressor. That’s provable. The live question is whether “therefore it’s nothing special” follows.

Your brain is a compressor too, in the relevant sense. You don’t store every moment you’ve lived. You extract patterns, keep the rules, and reconstruct on demand. That’s why eyewitness testimony is so unreliable — memory is reconstruction, not playback. Nobody concludes from this that human intelligence is just compression.

Compression isn’t all of intelligence. But intelligence requires it. The ability to find regularity in the world and store it in compact form deserves the name as much as anything else does.

One thing worth noticing: the heat has gone out of this fight. The 2023-2024 stretch produced a flood of papers and threads. Now it’s quiet. Not because anyone won — because LLMs started calling tools, executing code, and grinding through multi-day tasks. “Is this real intelligence” turns out to be a question you never need to answer to ship anything.

The takeaway

The math doesn’t move. Compression and prediction are the same operation, and LLMs compress circles around gzip. Whether that’s evidence they’re profound or evidence they’re mundane depends entirely on what you decided intelligence meant before you walked in.

Try it if you’re curious. A small local model and an arithmetic coder is all it takes. Run your own writing through it and the compression ratio becomes a score for how much of your prose the model already saw coming. Text that compresses well is predictable. Text that resists is new. How many bits is your writing worth?