4 posts tagged vllm.
Three runs of a self-hosted INT8 checkpoint on a patched engine average 77.0 — above the rented llama.cpp anchors and above Claude Sonnet 5, at about a dollar of electricity a run. The same weights scored 64.5 before the sampler was fixed.
vLLM's Triton top-k/top-p kernel drops the weakest surviving candidate when min_p, top_k and top_p are used together. Nine days of runs decoded near-greedily, every correctness instrument passed, and the investigation started on the weights.
A tensor-by-tensor comparison of the Unsloth Q8_0 GGUF and the official FP8 checkpoint against the BF16 weights, three purpose-built INT8 checkpoints to close the gap, and a corpus measurement that cannot resolve any of them.
The same model scored 11 points lower on vLLM's Anthropic bridge than on llama.cpp. The bridge hands the model its own prior reasoning back, it imitates itself, and within a few hundred steps 40% of its thinking is one 62-character sentence.