4 posts tagged methodology.
vLLM's Triton top-k/top-p kernel drops the weakest surviving candidate when min_p, top_k and top_p are used together. Nine days of runs decoded near-greedily, every correctness instrument passed, and the investigation started on the weights.
A tensor-by-tensor comparison of the Unsloth Q8_0 GGUF and the official FP8 checkpoint against the BF16 weights, three purpose-built INT8 checkpoints to close the gap, and a corpus measurement that cannot resolve any of them.
Nine model configurations — two self-hosted open-weights, four Claude generations, GPT-5.6 Sol, DeepSeek and GLM — each ran the same long-horizon enrichment pipeline three times, scored against adjudicated ground truth with weights fixed before results.
Two configurations swapped rank when third replicas landed — one of them spanning 15.8 points between identical runs. Single-run model comparisons measure the dice.