5 posts tagged evals.
Driving Chrome in an agentic loop, Qwen 3.8 Flash-Next navigates to a signed Alibaba Cloud object-storage URL it invented: 235 times in 17,679 navigations. Under identical workloads and prompts, Claude, GPT-5.6 Sol, GLM-5.3 and DeepSeek do not. It also reports sending such URLs when it did not.
Three runs of a self-hosted INT8 checkpoint on a patched engine average 77.0 — above the rented llama.cpp anchors and above Claude Sonnet 5, at about a dollar of electricity a run. The same weights scored 64.5 before the sampler was fixed.
Nine model configurations — two self-hosted open-weights, four Claude generations, GPT-5.6 Sol, DeepSeek and GLM — each ran the same long-horizon enrichment pipeline three times, scored against adjudicated ground truth with weights fixed before results.
Two configurations swapped rank when third replicas landed — one of them spanning 15.8 points between identical runs. Single-run model comparisons measure the dice.
Identical weights ranked last in one agent stack and first in another. Most leaderboard deltas I measured live in the plumbing, not the parameters.