Google Says It Can Run AI on Data It Never Decrypts. Here's What That Actually Means.

There’s one uncomfortable truth about cloud AI that nobody has solved: no matter how good your TLS is, your data eventually sits in plaintext in someone else’s server memory. Encrypted in transit, encrypted at rest, wide open during the part that actually matters. Google is now claiming it can close that gap — running model inference on ciphertext, without ever decrypting it. That’s fully homomorphic encryption, a technique that spent two decades filed under “theoretically beautiful, practically useless.”

Worth saying up front: this isn’t a breaking-news cycle. There’s no fresh wave of HN threads or a viral benchmark making the rounds. What’s here is a slow-building infrastructure story, and the interesting question isn’t what people are saying about it — it’s how much of the claim survives contact with the math.

Why FHE Took Twenty Years to Get Interesting

Standard encryption protects data at two of three stages. In transit: solved. At rest: solved. In use: you decrypt, you compute, you re-encrypt. That decryption window is where every cloud privacy promise quietly becomes a trust exercise.

Homomorphic encryption fills the gap. You add and multiply directly on ciphertext, and when you decrypt the result later, you get the same answer you’d have gotten computing on plaintext. The server never sees the input. It never sees the output either.

The catch was always cost. When Craig Gentry published the first working FHE construction in 2009, the overhead was widely cited at around a trillion times slower than plaintext computation. A one-second operation becomes roughly 30,000 years. Algorithmic progress since then has been genuinely impressive — but the gap is still measured in orders of magnitude, not in “a few times slower.”

What Google Has Actually Shipped

Google has been building this out quietly for years. It open-sourced a transpiler that converts ordinary C++ into FHE circuits. It shipped Private Set Intersection, letting two parties compute the overlap between their datasets without either revealing its contents. More recently, the whole stack moved onto MLIR-based compiler infrastructure — the same foundation used across modern ML toolchains, and a strong signal that hardware accelerators are in the plan.

The strategic direction is unmistakable. Google wants to take a technique that only cryptographers could touch and turn it into something a regular developer compiles. That matters more than it sounds. FHE’s biggest barrier was never purely performance — it was that maybe a few hundred people on the planet could implement it correctly. Compilers fix that class of problem.

But “AI inference” is doing a lot of work in the headline. Logistic regression on ciphertext? Been working for years. Small convolutional networks? Also working. That’s the tier where most published FHE-ML benchmarks live. It is not the tier most people picture when they hear an AI company say inference.

Where the Asterisks Start Piling Up

FHE is good at addition and multiplication. Transformers are largely built out of neither.

ReLU is a comparison, which is exactly what FHE handles badly. The standard workaround is polynomial approximation — and every degree you add buys accuracy while costing speed. Softmax needs exponentials and division. Layer normalization needs a square root. Each one is another approximation, another accuracy tradeoff, another performance tax.

Then there’s bootstrapping. FHE ciphertexts carry noise that grows with every operation. Cross a threshold and decryption fails permanently, so you periodically run a noise-reset — the single most expensive operation in the entire scheme. In a model with dozens of stacked layers, you’re paying that cost over and over.

So the honest summary: small classifiers work today. Mid-size models work under constrained conditions. Running a tens-of-billions-parameter LLM on encrypted data in real time is not something anyone has publicly demonstrated.

The Three Questions That Come Up Every Time

Every FHE announcement draws the same skepticism from developers, and it clusters into three arguments.

How many times slower, exactly? Any claim of practicality without a multiplier and a hardware spec gets discounted immediately, and the field earned that reaction. FHE has a long history of benchmarks cherry-picked from favorable parameter sets.

The model is still on their server. FHE hides your input from the provider. It does nothing to tell you which model ran, what preprocessing happened, or whether the weights got swapped out last Tuesday. It’s half the privacy problem, solved very expensively.

TEEs are right there, and they’re cheap. Trusted execution environments give you near-zero overhead. Apple’s Private Cloud Compute is the highest-profile example — encrypted memory, attestation, no plaintext exposure to Apple’s own operators. Yes, TEEs require trusting a hardware vendor, and yes, Intel SGX and AMD SEV both have side-channel research against them. But given a choice between a four-figure performance penalty and trusting Intel, most enterprises pick Intel.

That’s the real fault line: TEEs ask you to trust hardware, FHE asks you to trust math. Math is the stronger guarantee. It’s also, right now, the far more expensive one.

Why It Still Deserves Attention

The skeptical case is solid. But dismissing this entirely means betting against a trend line that’s been moving in one direction.

FHE has gotten meaningfully faster every year. Algorithms improved, and dedicated accelerator silicon is in development from multiple groups. GPUs are what dragged deep learning out of the lab; purpose-built FHE hardware could plausibly do the same thing here. Google laying compiler infrastructure years ahead of that hardware looks less like a research hobby and more like positioning.

Regulation is the other forcing function. In healthcare and finance — HIPAA, GDPR, sector-specific data residency rules — “slow but legal” beats “fast but prohibited” every time. A hospital that legally cannot ship patient records to a cloud GPU has a different math problem than a startup optimizing latency. In those markets, 100x slower inference is still a product.

The Bottom Line

What Google has demonstrated is closer to “this works under specific conditions” than “cloud AI privacy is solved.” But the compiler and hardware groundwork is too deliberate to write off as marketing. This sits somewhere between overstated and genuinely real — not between overstated and fake.

The test is narrow and easy to apply. In the next announcement, look for three numbers together: which model, on which hardware, at what multiple of plaintext speed. When a company publishes all three in the same paragraph, that’s the inflection point. Until then, the more useful question is the one every CTO in a regulated industry is already asking themselves: how much inference speed would you trade to make sure nobody else ever sees the data? Wherever that answer lands is exactly how big this market gets.