A One-Trillion-Parameter Model in 29GB of RAM — At Half a Token per Second

If you spend any time in local-LLM circles, you’ve seen the screenshot. Someone loads a trillion-parameter open-weight model on a machine that has no business running it, posts the terminal output, and the replies pile up. Then you look closer at the numbers: 0.5 tokens per second. That’s roughly a minute per sentence. Half the thread calls it the arrival of local AI. The other half calls it a stunt. Both camps have a point, and the interesting part is figuring out which one is measuring the right thing.

One caveat up front: this isn’t a roundup of community reaction, because there isn’t much to round up. Threads on this specific setup have been thin over the past month. So this is a look at the mechanics — why a trillion parameters can fit in 29GB at all, and whether the result is useful to anyone.

What 29GB actually means

Start with arithmetic. A trillion parameters at full precision — FP16, two bytes each — is 2TB. That’s not a desktop problem. That’s not even a single-server problem. So when someone says the model runs in 29GB, two things have happened in between.

The first is quantization. Instead of storing each parameter in 16 bits, you store it in 4, or 2, or in the more aggressive schemes, around 1.58. It’s JPEG for weights: some fidelity goes away, the file gets dramatically smaller. Push to 2-bit and 2TB becomes roughly 250GB. Better. Still not 29GB.

The second is disk offloading, and this is where the number gets slippery. The model doesn’t live in RAM. It lives on the SSD, and only the slice needed for the current computation gets pulled into memory. So 29GB isn’t the model’s size — it’s the peak working set. Everything else is streaming off disk, continuously. The model didn’t get smaller. It got sliced and fed through a narrow opening, one piece at a time.

That distinction matters more than any headline number, because it tells you exactly where the performance is going to die.

None of this works without Mixture of Experts

There’s a structural reason this trick is viable now and wasn’t a few years ago. Most large open-weight models today use Mixture of Experts architectures. The parameter count is a trillion, but processing any single token activates only a fraction of it — typically 3 to 5 percent.

Think of a restaurant kitchen with 100 cooks on staff. An order comes in, three of them move, the other 97 stand still. You don’t need all 100 at the stations. Keep them in the break room and call the ones you need. The stations are RAM. The break room is your SSD.

That’s what makes offloading tractable. In a dense model, every token requires reading every parameter, which means streaming the full 2TB off disk for each token, over and over. That isn’t slow — it’s arithmetically hopeless. MoE is the only reason the conversation happens at all.

Where the half-second-per-token goes

The bottleneck isn’t compute. It’s bandwidth.

Modern GPU memory bandwidth runs from several hundred GB/s past 1TB/s. A good NVMe SSD tops out around 7GB/s. That’s a gap of more than 100x. The GPU finishes its work and then sits idle, waiting for the next expert’s weights to crawl in from storage. This is the memory wall — the long-running reality that process improvements don’t help when the pipe moving data is the constraint.

In practical terms: a paragraph-length answer is maybe 400–500 tokens, so about 15 minutes. Ask for a code snippet and go make coffee. Interactive back-and-forth is off the table entirely. And if it’s a reasoning model burning thousands of tokens on its own chain of thought before it says anything, you can be waiting over an hour for the first visible output.

Worth noting one secondary cost: offloading is read-dominated, so write wear isn’t the main concern. But once contexts get long and the OS starts swapping, that changes. If you’ve been running this way for months, it’s a reasonable moment to check your drive’s health stats.

Why people keep doing it anyway

On throughput alone, this looks pointless. It isn’t, for three reasons.

Access is the achievement. “You need eight H100s” and “it’s slow, but it runs on a gaming PC” are not the same sentence. The second one means a grad student, an independent researcher, or a curious engineer can pull apart a frontier-class open-weight model and watch it work. Reading a benchmark table and running the thing on hardware you own produce very different kinds of understanding.

Batch work doesn’t care about latency. You can’t chat with it. But classify or summarize a few hundred documents overnight? Eight hours at half a token per second is roughly 14,000 tokens of output — no supervision required. For teams where cloud API spend is a real line item, or where the data legally cannot leave the building, a slow local model isn’t the worse option. It’s the only option. Healthcare and legal shops run into this constantly.

The bottleneck is moving in the right direction. DDR5 bandwidth keeps climbing. Apple Silicon’s unified memory approach turns raw RAM capacity into the selling point, and machines with 128GB or more of shared memory sidestep the disk entirely. PCIe 5.0 SSDs are becoming standard. If 0.5 tokens per second becomes 5 in two years, the practicality argument doesn’t get refined — it gets reversed.

The honest read

This is a proof of possibility, not a tool. If someone asks whether they should build a workflow on it today, the answer is almost always no. A 30B-class model quantized to 4-bit on 32GB of RAM, pushing 20 tokens per second, is a vastly better use of the same machine.

But the trajectory is worth tracking. Running a 70B model on consumer hardware was a punchline not long ago; now it happens on laptops. A trillion parameters could follow the same curve. The current 0.5 tokens per second isn’t a destination — it’s a marker someone planted at the trailhead.

So here’s the question that actually splits people: 100 tokens per second from the cloud with an instant answer, or 15 minutes on your own machine with data that never crosses your network boundary. If your answer changes depending on what you’re working on, that ambivalence is exactly why the local LLM scene refuses to go away.