Your Local LLM Isn't Dumb. Your Settings Are.

You finally splurged on the GPU. You pulled down the open-weights model everyone’s been posting about. And by the third turn of conversation, it’s forgotten what you asked and started confidently making things up. Meanwhile the benchmark charts say this thing trades blows with the frontier APIs.

Here’s the uncomfortable part: the model is probably fine. This is one of the oldest recurring stories in the local LLM scene — the vast majority of “my local model is stupid” complaints trace back not to the weights, but to the configuration wrapped around them.

A note up front: this isn’t a report on a specific viral thread. There wasn’t much fresh community chatter on this in the last 30 days. So instead of citing upvote counts, this is a rundown of the technical failure modes that have been litigated and re-litigated in the local inference community for years.

Suspect one: quantization shrank the file and the brain with it

Running a model locally almost always means running a quantized model. The original weights store each parameter in 16 bits. The GGUF file you downloaded crushed that down to 4 or 5 bits, cutting the size by two-thirds or more. That compression is not lossless.

The community consensus is that Q4 sits near the sweet spot — the point where you get most of the size savings before quality falls off a cliff. Below that, when you’re squeezing into limited VRAM at Q3 or Q2, things change. And they change deceptively. The prose still flows. But the model quietly mistypes a variable name, drops a digit in arithmetic, or skips an entire step in a chain of reasoning. Low-bit quantization doesn’t fail loudly. That’s what makes it dangerous.

There’s a piece of folk wisdom worth revisiting here: “a big model heavily quantized beats a small model lightly quantized.” Broadly true — until you go below 3 bits, where it frequently inverts. A 32B at Q4 will outperform a 70B jammed into Q2 on real work more often than not. And modern quant schemes complicate this further: two files both labeled 4-bit can differ substantially depending on which layers got extra precision. Picking by the number in the filename alone will cost you.

Suspect two: sampler defaults trading accuracy for “creativity”

This is the most maddening one, because nothing is broken. One setting is just wrong for your use case.

An LLM picks the next token probabilistically. Temperature, top_p, and top_k control how adventurous that pick is. Most local runtimes ship with a default temperature somewhere between 0.7 and 0.8. That’s a reasonable setting for fiction. It’s close to sabotage for code, math, or factual lookup. Even when the correct token carries 90% of the probability mass, you’re rolling dice on the other 10% at every single step.

Worse, the right values are model-specific. Reasoning-tuned models often ship recommending temperature 0.6 with top_p 0.95. Some coding models explicitly tell you to run near zero. And runtimes frequently do not read those recommendations automatically — you have to transcribe them from the model card by hand. Skip that and you’re running the tool author’s generic default instead of the settings the model builders actually tuned against.

Repeat penalty deserves the same scrutiny. It exists to stop the model looping on itself, but push it past 1.1 and it starts suppressing the tokens code legitimately needs to repeat: closing brackets, indentation, recurring variable names. If your code blocks keep terminating in weird places, look here before you blame the model.

Suspect three: the chat template, where one wrong character makes it a different model

Instruction-tuned models were trained to receive conversations in a specific format, with special tokens marking where the system prompt ends and the user turn begins. Break that format and you’re feeding the model something it has never seen in training.

The symptoms are distinctive. It answers your question, then invents your next question and answers that too. Strange tags leak into the end of responses. It flatly ignores the system prompt. It’s easy to conclude the model has weak instruction-following. Usually the template is just misaligned.

Modern runtimes do read the template baked into the model file, so this should be a solved problem. Two things still break it. Community re-quants sometimes ship with wrong or missing template metadata. And users manually override the prompt format in a frontend, then forget they did. Reasoning models make it messier still — runtimes handle thinking-process tags inconsistently, so the same weights can behave differently depending on where you run them.

Suspect four: your 128K model is actually running at 4K

This one tends to land hardest. The model card advertises 128K tokens of context. The runtime’s default context setting is often still sitting at 2048 or 4096. It’s there as a guardrail against out-of-memory crashes, which is defensible — the problem is that nothing warns you.

Paste a long document into that setup and the front of it gets silently truncated. No error. The model dutifully answers based on the half it can see. From your side it looks like the model read the whole file and summarized a fraction of it. That “suddenly forgot what we discussed earlier” behavior in long chats usually comes from the same place.

One caveat: maxing out context isn’t automatically the fix. The KV cache eats VRAM, and if it pushes model layers off the GPU and into system RAM, throughput drops by multiples. Set it comfortably above your actual working length, not at the ceiling.

The checklist

Work through it in order and most cases resolve. Start by checking what your context length is actually set to — not what the model supports. Then transcribe the recommended sampler values from the model card. For code or factual work, drop temperature to 0.2 or below and see what changes. If the same prompt produces wildly different results across two runtimes, suspect the chat template. And if you’re running Q3 or lower, benchmark it head-to-head against the next size down at Q4.

A local model isn’t a thing you download and finish. There are at least four configuration layers sitting between the file and you, and their defaults are tuned for safety and generality — not for your task. The commercial APIs feel better partly because someone else already did this tuning for you. Before you go shopping for different weights, open the settings panel. There’s more performance stranded in there than you’d expect.