The Sampler That Deleted the Runner-Up

Every vLLM run served from this box between 18 September and 27 September sampled near-greedily. Not because of the weights, the quantization, or my patches to the model code. vLLM’s Triton top-k/top-p kernel drops the weakest surviving candidate whenever a request uses top_k, top_p < 1 and min_p together and min_p has left between two and four candidates standing — which, on ordinary fluent text, is most positions.

With two candidates, the runner-up vanishes. A two-way choice became a coin with one side.

How it presented

The hybrid INT8 checkpoint went into production on 26 September and scored near the bottom of the eval — worse than both of the checkpoints it was built from. That is a strange result for a checkpoint that is, tensor for tensor, at least as close to BF16 as either source. So the first suspicion was the weights, and the first three tests were built to interrogate them.

All three passed, and that is what redirected the search.

• Decode-versus-prefill consistency. Greedy-generate tokens, re-score them through the prefill path, compare log-probs. Top-1 flips 3.3% / 3.3% / 0.4% / 3.8% across four context depths, p99 |Δlogprob| 0.25–0.47 — at the FP8 baseline, within noise. (A first 160-token run looked elevated at 10%; it was a small-sample artefact and the 240-token re-run settled it.)
• Corpus NLL under conditions. Two documents scored solo, on a prefix-cache hit, concurrently with a long generation, and solo again: every delta within ±0.0063 NLL of the reference. The forward pass and the disk-backed PLE path are fine.
• Config and launch parity. Every config file sha256-identical between the hybrid and FP8 checkpoints; the launchers differ only in pipeline-parallel size. MTP acceptance comparable (63.1% against 60.3%).

The weights were clean. So was the forward pass. What was left was what happened to the logits after the model produced them.

The test that found it

Take the model’s own top-20 log-probs at a fixed position from prompt_logprobs — the prefill path, which never touches the sampler. Apply the server’s sampling defaults offline. Then draw 300 actual samples from the live server at that same position and compare the histograms.

With two candidates the runner-up disappears. With three or four the last one is suppressed. With five or more the histogram matches. The pattern is the defect.

Bisecting the sampling parameters made it exact:

Every single filter is right. Every pair is right. Only the triple collapses — at both temperatures, on both the plain sampler and the speculative-decoding verify path.

Root cause

vLLM applies sampling in order: temperature, then min_p, then top_k/top_p. min_p sets everything below max + ln(min_p) to −∞, which turns a dense 248,320-entry logit row into one with two to four finite values in it.

In the Triton kernel, a row with fewer finite logits than k makes top-k a no-op, so the exact top-k-plus-top-p branch is skipped. The row then falls into a branch labelled standalone top-p sampling — and that branch does not sort. It estimates the nucleus cut-off from the mean and standard deviation of the finite logits in the first 1,024-entry block and a normal-CDF lookup table, then refines with a pivot search.

That estimate assumes a large, roughly normal population of logits. With two to four finite values it is simply wrong, and the lowest survivor gets masked. The branch’s own comments say it exists to handle grammar masks; it was not written with min_p in mind.

Reproduced standalone against vLLM’s own PyTorch reference, no server involved:

For two finite logits the second is dropped whatever its mass — tested from 5% up to 49%. A token the model rated nearly a coin-flip was deleted.

There is one more wrinkle that explains the bisect exactly: when top_k is None, the dispatcher takes a different, split-row code path that is correct. That is why top_p + min_p passed and top_k + top_p + min_p did not.

Why nine days of instruments missed it

This is the part worth carrying away.

The defect produces fluent, plausible text — it is precisely what greedy decoding produces. It never errors. Nothing in a log looks wrong. And every correctness instrument in the project was blind to it by construction:

• The decode-versus-prefill check runs at temperature 0. A greedy test cannot validate a sampler. It proved the logits were right and said nothing about what was done with them.
• Every NLL and log-prob measurement goes through prompt_logprobs, which is the prefill path. The sampler is not in it.
• The verification written for an earlier patch — the one that allowed min_p under speculative decoding — checked that no sampled token fell below the min_p threshold. It did. The failure was in what happened to the tokens min_p kept, one stage later, and only when two other filters were also on. Check filter combinations, not filters.

The instrument that works is cheap: compare the sampled histogram at a fixed position against the model’s own prompt_logprobs, with the filters applied offline. It runs against a live server in a minute. There is now a standing rule to run it after every vLLM upgrade — the server-defaults row must match the single-filter rows, and never read 100/0.

The fix

The dispatcher now always uses the exact PyTorch path — sort, cumulative sum, mask — with the Triton kernel reachable only behind an environment variable. The cost is 2.04 ms per call for the largest batch this server produces, against a ~10 ms decode step.

Verified offline against the reference for 2, 3, 4, 5, 8, 20 and 64 finite logits and for dense rows, then live after the restart:

Upstream has a fix in flight for a different defect in the same file. The few-finite-logits case does not appear to have been reported, and should be.

What this invalidates

Two posts on this site draw on runs that were sampled with this defect. Being specific about what survives matters more than a blanket retraction.

Unaffected, and stated with confidence: every llama.cpp run, which uses its own sampler; every vLLM run before 18 September, when there was no min_p override and the rows were dense; every greedy or temperature-0 measurement; every prompt_logprobs and NLL number; every throughput figure; and the entire per-tensor fidelity study, which reads weights off disk and never runs the model.

So the weight-fidelity post stands on its substance. Q8_0 really is ~5× closer to BF16 than FP8 where both quantize, the hybrid build really is bit-identical to the GGUF on experts and exact everywhere else, and the corpus log-probs really cannot tell any of them apart. None of those numbers touch a sampler. What is confounded is its framing: it opens by saying transport repairs took vLLM to within 1.7 points of the llama.cpp anchors and closes by calling those 1.7 points unexplained. They now have a candidate explanation, and it is this.

The carried-thinking post is more mixed. Its mechanism survives intact, because the evidence for it was never a score: the rendered-prompt diff, the parity harness, the filler-sentence counts of 327 against 6, and the within-segment decay curves are all transcript and prompt measurements. The controlled comparison — same server, same defect, only the transport differing — is still controlled.

Its scores are not usable. 64.3, 74.0 and the comparison against the llama.cpp anchors’ 75.7 all pit a near-greedy vLLM against a correctly-sampled llama.cpp. Worse, the two defects point the same way: near-greedy decoding independently promotes repetition, and Qwen’s own guidance is not to greedy-decode its thinking models. So the sampler defect plausibly amplified the filler-sentence collapse it was measuring. The mechanism is real. Its magnitude is not established, and neither is how much of the vLLM-versus-llama.cpp gap it ever explained.

The honest statement is that the hybrid’s eval score is void, the ordering between the hybrid and FP8 is unestablished — both were sampled with the defect, so the gap between them is not explained by it either — and everything needs a clean re-run.

The first clean re-run

The first eval run on the patched server has finished. Same INT8 hybrid weights, same client path, same template, same effort setting — the only thing that changed since the last run on these weights is the sampler.

75.0 out of 100, with zero tier-2 errors: no wrong site, no wrong link, no false flag, no fabricated provenance. That is the first run on this box, on any engine, to manage that. The designed acquisition trap — a national brand implied by search results, worth 6 points each time — was declined; it had been taken in seven of eight FP8 runs and in both pre-patch runs on these same weights.

That is +10.5 points over the pre-patch mean on identical weights, and the two rows at the bottom are why I believe it is the sampler rather than something else. Reasoning depth did not move. The model is not thinking longer or harder; it is making different decisions at the moments where it was previously handed a coin with one side. Every point between 59.8 and 75.0 is a judgement call that came out differently — the acquisition trap, the bill-pay-versus- ordering flags, the directory blurbs, the dedupe merges. Fabrication was clean in all three.

For scale, against the configurations that have three runs each:

A self-hosted model on cards in a tower, 0.7 points under the rented anchors, for about C$0.80 of electricity against $17–41 to rent the same work and $55–85 to buy it from an API.

Three caveats, and the first was the big one. This was n=1 when first published, and a single run that lands where you hoped is exactly the result to distrust. The two replicas have since finished: 75.4 and 80.6, for a mean of 77.0 across three runs — above the llama.cpp anchors rather than just level with them. That series, and what it settles, is its own post.

Second, two of the three launches ended with a thinking-and-text turn and no tool call — the end-of-turn signature from the carried-thinking post. Both resumed with state intact and are non-scoring by the standing rule, but two in one run is new for this box and wants watching against the patch.

Third, what this does not settle: the ordering between the hybrid and FP8 checkpoints. Both of the numbers that would decide it were taken with the defect, so the gap between them was never explained by it. That still needs its own clean pair of runs.

What I would do differently

The configuration change that triggered this was correct. Setting min_p 0.05 on vLLM to match llama.cpp’s default was the right call, made for the right reason, and it stays. It was the trigger only because the kernel is wrong.

But the investigation spent one day on chat templates and reasoning effort, and the first hours of another on weights, before anyone checked what the sampler actually emitted. The order should have been the other way around. When a quality gap appears after a configuration change, test that change’s downstream effects before you re-examine the model. A fast path that is correct for dense inputs can be wrong for sparse ones, and min_p is a machine for producing sparse ones.