Attention Was the Last Sacred Cow. Kimi Linear Just Put It on the Table

Every model launch now comes with a context window number attached. A million tokens. Two million. Then you actually use it, and things get slow. Expensive. And somewhere past the halfway mark, the model starts acting like it forgot the beginning. That is not laziness. That is architecture. And the part of the architecture responsible — attention — is finally getting renovated. Kimi Linear is the most interesting thing to come out of that renovation so far.

Worth noting up front: this isn’t a story that’s lighting up Hacker News or Reddit. Search the last month of discussion and you’ll find almost nothing. It’s the other kind of story — the one that quietly rearranges the board while everyone argues about benchmarks.

Why Attention Costs So Much

The mechanism is almost embarrassingly simple. Every token in the sequence looks at every other token, once. That’s it. That’s why transformers capture context so well.

The problem is that “everyone looks at everyone” scales quadratically. A thousand tokens means a million comparisons. A hundred thousand tokens means ten billion. You made the input 100x longer and the compute bill went up 10,000x.

Then generation adds a second problem: the KV cache. Every token you’ve already processed has to stay in memory, and that memory grows linearly with context length. So the costs come at you from two directions at once — compute climbing quadratically, memory climbing linearly. GPU memory is finite, so at some point you shrink the batch size. Throughput drops. Cost per token goes up. That’s the invoice hiding behind “supports 1M tokens” on the spec sheet.

The Graveyard of Linear Attention

People have been trying to fix this for years. Linear attention research goes back well before the current wave, and the idea is clean: instead of remembering every token, compress everything into a fixed-size state and carry that forward.

The state never grows. Compute goes linear. The KV cache stops existing as a concept. It’s an RNN, essentially, but restructured so training can still be parallelized. Mamba, RWKV, GLA — all variations on this bet.

And they all paid the same tax. Squeeze arbitrary amounts of information into a fixed-size state and the information gets blurry. The failure mode shows up most brutally in retrieval: pull out that one specific number from twenty pages back. Average benchmark scores looked respectable. Hand these models a long document and something felt hollow. That’s why linear attention has spent years stuck in the “promising, but” category — the architectural equivalent of a prospect who never makes the majors.

Hybrid Beats Pure

Kimi Linear’s answer was to stop being a purist. Rather than going all-linear, it interleaves linear layers with standard full-attention layers at roughly a 3:1 ratio — three linear layers, then one full attention layer, repeat.

The reasoning holds up. Most layers are doing local pattern work: scanning nearby context, summarizing, compressing. A fixed-size state handles that fine. But a few layers need to reach across the sequence and grab one specific token precisely. Give those layers full attention. You end up with a division of labor at the layer level — cheap where cheap works, expensive only where precision actually matters.

The linear component itself is KDA, or Kimi Delta Attention. The key move is a delta-rule update. Instead of overwriting the state wholesale or just adding to it, KDA adjusts the state by the difference between what’s arriving and what’s already stored. On top of that sits fine-grained per-channel gating: how much to forget and how much to keep gets decided independently for each dimension.

The old approach was writing over the same sheet of paper again and again until it turned gray. KDA erases only what needs erasing and rewrites only what needs rewriting. Same sheet of paper, far less smudge.

The Numbers

Architecture stories are only interesting if the numbers land. A few here do.

KV cache drops by roughly 75%. That follows directly from the linear layers not needing one, but the practical implication is what matters: you can run close to 4x the batch size on the same GPU. In inference economics, that’s not a tweak. That’s a different cost structure.

At million-token decoding, throughput improves by around 6x. At short context, the gap is negligible. The advantage widens as the sequence grows, which is exactly the regime the design targets.

And quality didn’t regress. On several benchmarks, it beat comparable full-attention models outright. That’s the part worth sitting with. Every linear attention paper for years has essentially said “slightly worse, much faster.” This one didn’t need the caveat.

Benchmarks are still benchmarks, of course. Whether 3:1 is the right ratio for every workload, whether it holds up under genuinely adversarial retrieval — feed it an entire codebase and ask for one function signature — those are open questions. But the direction is no longer ambiguous.

Why This Reshapes Things

The industry’s center of gravity has shifted toward inference-time compute. Agents call tools in long loops. Models think through extended reasoning chains. Conversation histories get carried forward indefinitely. Every one of those trends spends context.

Which makes attention cost the constraint that increasingly decides who wins. We’re entering a stretch where running the same intelligence at a quarter of the price beats making the model marginally smarter. The recent examples of open-weight models matching frontier models on specific tasks fit the same pattern: the price gap is starting to matter more than the capability gap.

There’s a second thing worth noticing. None of this is visible to users. No UI change. No new feature. One day the context limit is higher, responses come back faster, and the price per million tokens dropped. Infrastructure improvements always arrive this quietly.

The Takeaway

The transformer paper is over eight years old. Model sizes changed, training data changed, post-training changed completely — and attention stayed basically as written. Kimi Linear is the signal that the last untouched piece is now negotiable.

The genuinely interesting part is that the winning answer wasn’t “replace attention with something new.” It was “mix the new thing with the old thing in the right proportion.” Hybrids beat purity. That’s usually how technology actually turns over — not a clean replacement, but a negotiated one.

So when your AI tool’s context limit quietly goes up in some future release, there’s a decent chance something like this is underneath it. It won’t make the changelog.