7 posts tagged local-inference.
Driving Chrome in an agentic loop, Qwen 3.8 Flash-Next navigates to a signed Alibaba Cloud object-storage URL it invented: 235 times in 17,679 navigations. Under identical workloads and prompts, Claude, GPT-5.6 Sol, GLM-5.3 and DeepSeek do not. It also reports sending such URLs when it did not.
Three runs of a self-hosted INT8 checkpoint on a patched engine average 77.0 — above the rented llama.cpp anchors and above Claude Sonnet 5, at about a dollar of electricity a run. The same weights scored 64.5 before the sampler was fixed.
vLLM's Triton top-k/top-p kernel drops the weakest surviving candidate when min_p, top_k and top_p are used together. Nine days of runs decoded near-greedily, every correctness instrument passed, and the investigation started on the weights.
A tensor-by-tensor comparison of the Unsloth Q8_0 GGUF and the official FP8 checkpoint against the BF16 weights, three purpose-built INT8 checkpoints to close the gap, and a corpus measurement that cannot resolve any of them.
The same model scored 11 points lower on vLLM's Anthropic bridge than on llama.cpp. The bridge hands the model its own prior reasoning back, it imitates itself, and within a few hundred steps 40% of its thinking is one 62-character sentence.
Nine model configurations — two self-hosted open-weights, four Claude generations, GPT-5.6 Sol, DeepSeek and GLM — each ran the same long-horizon enrichment pipeline three times, scored against adjudicated ground truth with weights fixed before results.
The capstone: the full pipeline on a 4090+3090 pair — faster than two of three rented datacenter runs, for 68 cents of electricity. Plus the +17% environment variable and the wall that stopped a bigger model.