Dust’s Backprop-Free Training Has a GPU-Bill Question

Key takeaways

• Dust aims to pretrain transformers without backpropagation.
• A Hacker News commenter highlighted promising larger-model results, but efficiency per candidate does not establish lower training costs.
• Easier parallelization could shorten training time even if total compute increases.
• The useful comparison is compute and elapsed time needed to reach the same model quality.

A faster training run can still leave you with a bigger GPU bill. That tension sits at the heart of Dust, an effort to pretrain transformers without backpropagation. An October 5, 2026 Hacker News thread about Dust drew 180 points and 43 comments, with discussion touching on the question that matters for anyone paying for training: what does this approach buy you?

Removing backpropagation leaves the hard part

Transformers are the neural-network architecture behind today’s language models. Pretraining teaches those models patterns from large amounts of data by adjusting their internal parameters.

The standard toolkit includes backpropagation, which calculates how parameters contribute to a model’s error. That information guides updates to the model.

Dust explores doing the pretraining without this step. The proposed change concerns how a transformer learns; it does not, by itself, mean replacing the transformer architecture.

But finding useful parameter values still takes work. Removing backpropagation only reduces costs if the replacement makes the overall training process more economical. A missing step on the diagram is not automatically a discount on the invoice.

The larger-model result needs a denominator

One observation in the Hacker News discussion is particularly interesting. A commenter highlighted a 243 million-parameter model that outperformed a model roughly 120 times smaller across most candidate population sizes.

Here, “population” means a group of candidate solutions. The commenter interpreted the result as evidence that larger models could use those candidates more effectively.

That is an intriguing reading. More parameters mean more numbers to tune, so a larger model doing better under these conditions deserves attention. Still, this is a community interpretation of experimental results, not proof that larger models are cheaper to train.

The denominator matters. Efficiency per candidate and efficiency per unit of compute measure different things. Evaluating one candidate for a larger model may require substantially more computation. Better results from fewer candidates could still come with a higher total bill.

A useful comparison with backpropagation would hold model size, training data, and evaluation criteria constant. Then it would measure how much computation each approach needs to reach the same quality.

Parallelism can buy time without saving compute

Another view expressed in the discussion was that Dust appeared less computationally efficient than backpropagation but easier to parallelize. That is a proposed tradeoff, not an established cost comparison.

Consider a hypothetical training job that takes one GPU 100 hours. If it splits perfectly across ten GPUs, it finishes in ten hours.

Both runs consume 100 GPU-hours. One gets you an answer much sooner.

This is an illustration, not a Dust benchmark. Real systems also spend time coordinating work, communicating between devices, and collecting results. Those overheads can increase total resource use and limit the speedup.

Even so, finishing sooner can have value. A team waiting on a result before choosing its next experiment might willingly spend more compute to shorten that wait. A team with a fixed GPU allocation may care far more about how much useful training it gets from each hour of hardware time.

Dust’s practical appeal depends on both measurements: total compute consumed and elapsed time to the target quality.

Scaling means passing two separate tests

The first test is whether the training method continues to work well as models get larger. The comment about the 243 million-parameter model speaks to that question, but it cannot establish what happens at much larger sizes.

The second test is whether adding hardware produces a worthwhile speedup. If candidate evaluations can run independently, that could help. The actual benefit would still depend on communication costs, how work is distributed, and how quickly results can be combined.

The interesting possibility is that both effects hold together: larger models remain trainable, while additional hardware brings results back sooner. That could make the method useful even with higher total compute requirements. Each part needs evidence.

The discussion also included a hybrid proposal: start with a model pretrained through backpropagation, then continue training with Dust’s approach. That is an experiment someone suggested, not a demonstrated improvement.

Dust’s practical test is straightforward: reach a defined quality level, then compare the compute bill and the time spent waiting. If it needs more GPUs but delivers useful results sooner, the decision becomes how much that saved time is worth.

Sources

• Dust: Pretraining Transformers Without Backpropagation — Hacker News discussion