The Chain of Thought Was Never a Moat

Here’s the strange part about this story: almost nobody is arguing about it in public. Search the last month of Hacker News, Reddit, X — the discourse is thin to nonexistent. Which is remarkable, because the question underneath it may be the most consequential unsettled issue in commercial AI right now. Are the reasoning traces that frontier labs hide from you actually defensible property? Or were they always a curtain rather than a wall?

Why the labs pulled the curtain

When OpenAI shipped o1 in late 2024, users noticed the asymmetry within hours. The model thinks. You don’t get to see it think. What you get is a sanitized summary. The raw chain of thought never appears in the API response — but it does appear on your bill.

The labs gave two justifications. The first was safety, and it’s genuinely reasonable. Unfiltered reasoning is where misalignment shows up first: a model rationalizing a bad answer, planning a deception, drifting off-policy. If you expose that stream to users, you inevitably start optimizing it to look good, and the moment you train the reasoning to be presentable is the moment it stops being a useful monitoring signal. Anthropic and OpenAI researchers have both made versions of this argument in public, and it holds up.

The second justification is commercial, and nobody pretends otherwise. Reasoning traces are premium distillation feedstock. A dataset of final answers teaches a small model to mimic outputs. A dataset of full reasoning chains teaches it to mimic the process — a dramatically shorter path from a 7B model to something that punches at 70B. When DeepSeek released R1 in January 2025 and the industry lost its mind, OpenAI’s response was to say it had evidence of distillation from its outputs. That was the moment the chain of thought stopped being a safety artifact and became a trade secret.

The curtain turned out to be thin

Hiding something is not the same as deleting it. Hidden reasoning still shapes the visible output, and that’s exactly where researchers started prying.

The most direct approach is logit analysis. When an API returns log probabilities, it leaks internal state. Treat the full response as a probability distribution and you can start reasoning backward about which intermediate paths the model took. This isn’t theoretical: in 2024, a team including Google DeepMind researchers published a model-stealing attack that recovered the embedding dimension and part of the final-layer projection matrix of production GPT models — through the API alone, for a few hundred dollars in credits. They weren’t reconstructing the whole model. But the premise that a black-box API is inherently safe died that day.

Second: timing and token counts. The labs hide the text of the reasoning but publish the reasoning token count, because they have to bill you. That’s a side channel. Fire a few thousand variants of the same prompt and measure how each perturbation moves the reasoning token count, and you get a map of where the model hesitates and where it’s confident. You don’t learn the content. You learn the shape — and shape is often enough.

Third, and most practical: just ask. The summaries already carry a substantial fraction of the underlying reasoning. Add “show every intermediate calculation” or “verify each step as you go” to the prompt and the model rewrites much of its hidden reasoning into the visible answer, because that’s what it was trained to do when asked. This isn’t an attack. It’s reading the documentation.

Distillation didn’t need an attack

Now the uncomfortable part. Even without any of the above, distillation was already happening at scale.

It’s an open secret that a large share of the open models released since 2023 were trained partly on outputs from frontier models. The barrier to entry is API credits and patience. Reasoning models made it easier, not harder, because in verifiable domains — math, code, formal logic — you don’t need the trace at all. Ask the strong model hard questions, keep only the responses that pass the checker, train on those. Rejection sampling transfers a startling amount of capability with no chain of thought required.

So hiding the traces never prevented distillation. It made it more expensive. That’s a speed bump, not a moat. And once you frame it that way, the labs’ next moves become obvious: tighter rate limits, anomaly detection on usage patterns, ID verification for high-volume tiers, explicit anti-distillation clauses in the terms of service. When the technical defense doesn’t hold, you fall back to contracts and infrastructure.

Security research or IP theft?

This is where it gets genuinely contested.

The labs’ position is clean. We spent hundreds of millions of dollars on RL infrastructure, compute, and human data. You spent forty thousand on API calls and copied the result. Our terms prohibit using outputs to develop competing models. That’s free-riding, full stop.

The counterargument is not weak. These same labs built their models by ingesting the entire web without asking anyone. The organizations arguing hardest that training on copyrighted data is transformative fair use are also arguing that their own outputs are proprietary. You can hold both positions, but you have to do some work to explain why. And there’s a legitimate research interest here: documenting what an API leaks is standard security practice. Undisclosed vulnerabilities don’t get fixed.

Then there’s auditability, which is the argument I find hardest to dismiss. If a model is being used for medical triage or legal analysis, and no one — not the user, not a regulator, not a court — can inspect how it reached its conclusion, that opacity is itself a risk. The EU AI Act’s transparency obligations for high-risk systems were not written with hidden chains of thought in mind, but they’re on a collision course with them. Meanwhile the user pays for tokens they are not permitted to read.

My read: this isn’t a case where one side is right. It’s a case where the same act has two names. A university lab does it and publishes a paper — security research. A competitor does it quietly — IP theft. The methodology is identical. The difference is disclosure and intent, and intent is notoriously hard to prove in court. When this eventually gets litigated, it will be ugly.

What actually changes

A few things are already visible.

APIs will get less transparent. Fewer logprobs, rounded token counts, jittered response timing. The cost lands on legitimate users: the developer debugging why a prompt regresses at temperature 0 loses a diagnostic tool, and gains nothing.

Open weights benefit by contrast. Models that ship their full reasoning — DeepSeek’s R1 line, Qwen’s reasoning models, the open reasoning efforts coming out of Allen AI and elsewhere — are winning developer trust precisely because you can audit them, fine-tune on them, and predict them. Every defensive measure a closed lab adds sharpens that contrast.

And one more, which I think is underrated. The real winner of this fight may be open reasoning datasets. High-quality chain-of-thought corpora that anyone can use, released deliberately rather than extracted, remove the economic motive for the attack entirely. That work is already underway, and it’s the only resolution here that doesn’t require a courtroom.

What was actually protected

The hidden chain of thought was never a moat. It raised the cost of entry modestly, and in exchange, users lost the ability to inspect computation they paid for. Whether the labs were protecting safety or protecting margin is a judgment each of us can make.

If you can’t see why your AI reached its answer, how much should you trust the answer? And if the opacity is defended on safety grounds by the same party that profits from it, what would it take to tell the difference?