The Quantization Was Never the Cost

Three replicas of a self-hosted INT8 checkpoint on the patched engine: 75.0, 75.4, 80.6 — mean 77.0, spread 5.6. That is above the llama.cpp anchors at 75.7, above Claude Sonnet 5 at 76.7, and second in the entire field only to Opus 5.5.

It ran on two mining cards and a gaming GPU in a tower, for about a dollar of electricity a run.

What the fix was worth

For six weeks the working theory was that self-hosted quantized weights were simply worse than what the rented llama.cpp anchors were serving. Two investigations chased that: one found a transport defect feeding the model its own reasoning back, another measured every tensor against the official BF16 release. Neither closed the gap, and the second found that the weights were, if anything, better than what the anchors ran.

Then the sampler turned out to be deleting the runner-up token on most positions. Here is what happened to the same checkpoint across five runs:

+12.5 points on identical weights, identical hardware, identical client path. The only thing that changed is that the engine stopped throwing away the second-most-likely token.

The two bottom rows are why I trust the attribution. Reasoning depth did not move. The model is not thinking longer or harder on the patched engine — it is making different decisions at the positions where it was previously handed a coin with one side. Every point between 64.5 and 77.0 is a judgement call that came out differently: the acquisition trap, the bill-pay-versus-ordering flags, the directory blurbs, the dedupe merges. Fabrication was clean in all five.

The weights question is closed

This is the result that matters beyond one box.

The self-hosted checkpoint and the rented anchors are, in the parts that carry the model, the same weights — the hybrid build’s routed experts and lookup table are byte-identical to the GGUF the anchors served, with the dense tensors kept at exact BF16 rather than quantized. Six weeks of work went into establishing that they were not the problem, and two separate investigations failed to explain why they nevertheless scored six points lower.

They no longer do. Same weights, different engine, and the engine that was losing now wins by 1.3 points. The quantization was never the cost. The engine was — first a transport that fed the model its own reasoning, then a sampler that deleted its second choice.

There is a lesson in the sequence, and it is not flattering. Every step of the investigation was reasonable in isolation: a scoring gap appeared, the weights were the obvious difference between the two stacks, so the weights got measured — exhaustively, 76,488 tensors, twice. The answer was in neither checkpoint. It was in the code between the weights and the output, which nobody measures because it is not supposed to be a variable.

The clean run

The third replica scored 80.6, and it is the first run in the entire evaluation — any model, any engine, any provider — with zero tier-2 errors, zero tier-3 errors, zero false flags, and the designed acquisition trap declined. Every point it lost was a blank link, an objective territory check, a category-only exclusion, or a flag it read too shallowly. Nothing it published was wrong.

Across all three patched runs the tier-2 count is zero: no wrong site, no wrong link, no false flag, no fabricated provenance. That category had never been empty for this configuration before.

What got worse

One behaviour regressed, and it would be dishonest to bury it. Text-only end-of-turn seams rose from zero to four across the three patched runs — sessions ending with a thinking-and-text turn and no tool call, the signature from the carried-thinking post. They are non-scoring, state resumes intact, and the anchors show the same failure (0 / 0 / 4 across their three runs), so it is not unique to this engine. But each one costs a full re-prefill, and it is the one number that moved the wrong way after the patch. It wants watching.

There was also one tier-3 error in the second replica: a hijacked domain published after an archive recovery wrote the live domain into the record, then shipped despite being flagged for manual review. That is a spec gap rather than a model failure — “deferred to manual review” rows keep shipping across every family — but it cost 6 points and it is the difference between that replica and the clean one.

Caveats

n=3 is the floor, not a comfort. The spread is 5.6 points, which is ordinary — the anchors span 5.4 and the Sonnet family 9.6 — but it means the 77.0 mean carries a real uncertainty and the 80.6 is one roll of the dice, not a new baseline.

Cost is not like-for-like. The self-hosted figure is modelled electricity in Canadian dollars for a machine that already exists; the API figures are billed US dollars. It excludes the hardware, which is the honest objection — though the rented-GPU row at $17–41 a run assumes no hardware at all and is still an order of magnitude above the electricity.

This configuration is off the main board by the project’s own rule: the reference infographic compares shipped checkpoints, and this is a checkpoint I built. It earns a row in this post, not on that chart.

And the patch is the finding. The engine’s version string did not change; only the process start time proves the restart happened. If you are running vLLM with min_p alongside top_k and top_p, the sampler post has the reproduction, the mechanism and the one-minute test that catches it. Twelve and a half points were sitting in a Triton kernel’s fast path for nine days, and every instrument on the box said the system was healthy.