REALLYCUTEPUPPIES.AI
Eval · Part II · evals · variance · methodology

My Leaderboard Flipped at n=3

Two configurations swapped rank when third replicas landed — one of them spanning 15.8 points between identical runs. Single-run model comparisons measure the dice.

September 7, 2026 · 3 min read

At two runs per configuration, my eval’s open-weights standings read: 27B in front, flash just behind, 74.5 to 74.4. I nearly published that. The third replicas rewrote it: flash rose to 75.7 while the 27B rolled a 62.6 — from a configuration identical down to the sampler — and fell to 70.5, below a Claude generation it had been “beating.”

606570758085Sonnet 5flash Q8Opus 5Opus 4.827B BF16Opus 4.6● runs | mean
Fig. 1 — Every scored replica, by configuration. The 27B's spread (15.8 points) is wider than the gap between the field's best and worst means. Opus 5's 2.6-point spread is the tightest — a genuinely marketable property that no single-run benchmark would ever surface.

Three observations that reframed how I read every model comparison now:

Replica spread dwarfs most config effects. Quant rungs measured under identical configs — 1.75-bit against 5-bit-class — differed by 0.4 points. The 27B’s replica spread was forty times that. I’ve watched people (myself included) agonize over quant selection while implicitly trusting n=1 results whose noise floor eats the whole decision space.

Variance is a model property, and it’s purchasable. Opus 5 held a 2.6-point spread across replicas; my flash config held 5.4; the 27B sprayed 15.8. If you’re running a production pipeline, the tight-variance model at a slightly lower mean may strictly dominate the lottery ticket — “sometimes rolls the best run in the field” is not a deployment property.

The dice touch everything downstream. Same weights, same config, same sealed inputs: one 27B run caught a designed trap that its own sibling fell for; one run omitted five records its sibling found. At temperature 0.7 over a multi-hour horizon, tiny early divergences compound into different research paths entirely. (No, dropping temperature isn’t the fix — these models’ own cards warn against greedy decoding for long reasoning, and cold sampling just measures the mode of a distribution you’ll never deploy at.)

The uncomfortable arithmetic: honest means need n≥3, which triples eval cost, which is why almost nobody does it — the six open-weights runs behind these numbers cost about $126 of rented GPU time in total — less than one run of the field’s most expensive API model. The alternative is publishing coin flips with axis labels.

Full per-configuration results, error breakdowns, and the economics live on the Gauntlet board.