3 posts tagged quantization.
Three runs of a self-hosted INT8 checkpoint on a patched engine average 77.0 — above the rented llama.cpp anchors and above Claude Sonnet 5, at about a dollar of electricity a run. The same weights scored 64.5 before the sampler was fixed.
A tensor-by-tensor comparison of the Unsloth Q8_0 GGUF and the official FP8 checkpoint against the BF16 weights, three purpose-built INT8 checkpoints to close the gap, and a corpus measurement that cannot resolve any of them.
A third of Qwen's flash model is a predictive n-gram table that never does a matmul. It quantizes as a step function, offloads to RAM for free, and even runs from SSD.