Self-Hosted vs Frontier: 27 Runs on a Real Agentic Pipeline

Most public model comparisons are short tasks with fresh contexts, run once. This is the opposite: one long-horizon agentic workload that runs unattended for hours, makes hundreds of judgment calls, drives a real browser, and gets scored against human-adjudicated ground truth. The question it was built to answer is whether open weights on hardware you control can do that work as well as a frontier API.

Nine configurations — two self-hosted open-weights models, four generations of Claude, GPT-5.6 Sol on a ChatGPT subscription, and two third-party APIs. Three runs each, 27 in total, on identical sealed inputs with error weights ratified before a single result existed.

The result in slot 2 is a self-hosted Q8 model finishing 1.0 point behind Claude Sonnet 5 on the mean. Before that becomes a headline, the honest reading: at n=3 that gap means nothing. Sonnet’s three runs span 71.6–81.2 and Flash’s span 73.0–78.4. They overlap almost completely. Anyone reporting “within 1 point” as a finding is reporting noise.

What does survive n=3 is the weaker but sturdier claim you can make from min-max separation — where one configuration’s worst run beats another’s best:

• Flash Q8 has the highest floor in the field, at 73.0. Higher than Sonnet 5’s 71.6, higher than every API configuration here.
• On that basis Flash Q8 separates cleanly from DeepSeek (max 71.4), GLM (max 72.4) and Opus 4.6 (max 70.4). It does not separate from Sonnet 5, GPT-5.6 Sol, Opus 5, Opus 4.8 or the 27B.
• No adjacent pair in the top four is separable. The ordering among them is not a result.

So the defensible version is this: a self-hosted open-weights model was not distinguishable from the best frontier configuration tested, and was the most reliable performer in the field by worst-case run. That is a smaller claim than the ranking implies, and it is the one the data supports.

The full board

Open-weights models were pinned to maximum reasoning effort through a verified proxy; GPT ran at xhigh and GLM at max, each confirmed by a reasoning-token probe before launch. DeepSeek has no effort levels to set — its thinking mode is binary. Both self-hosted rows were executed on rented GPUs (Flash on 2× RTX PRO 6000, the 27B on 1×), so the rented price is the measured one and the electricity figure is a model; both are unpacked below.

What the number actually means

A score is 100 minus the weighted sum of a run’s errors, and the weights encode a specific theory of harm rather than a generic notion of accuracy:

Those ratios were fixed before any run was scored, which matters more than their exact values. A leaderboard’s single number is always a policy about how you trade honesty against completeness — publish the exchange rate, or you are making that policy for your reader silently.

That policy has teeth, and two cases in this board show it. GPT-5.6 Sol carries the field’s worst territory-flag count (7.0) largely because it produced the field’s largest territory sets, at 10–12.5k rows per run — more output means more surface for a topology check to catch, so thoroughness is taxed. And the two bottom-tier APIs fail in opposite directions: DeepSeek under-claims flags it didn’t browse deeply enough to confirm, while GLM over-claims them at 2.7 per run, the field’s highest, reading “pay your bill online” as online ordering. Reweight under-claims against false assertions and those two swap.

A different rubric would produce a different board from the same 27 runs. Any eval that doesn’t hand you its weights is making that choice for you silently.

The workload, and why it resists gaming

300 frozen listings from a map platform — one category of local physical-service business in one US state, full of real noise: duplicates, look-alikes, defunct operators, franchise branches. Every run has to filter real providers from look-alikes, resolve brands and acquisitions into companies, verify each live website (catching dead, hijacked and builder-fake pages), enrich with evidence-cited attributes, map service territory to postal-code level, and publish relational CSVs behind an integrity gate.

Five hard rules do the heavy lifting:

• No code for judgments. Regex, fuzzy matching and similarity scores are forbidden as decision procedures; code is allowed only for mechanical I/O. A run cannot pattern-match its way to a score.
• One answer per field. Each judgment is made once and flows downward — corrected in place, never re-derived or forked.
• Read, don’t classify. Attributes must come from the provider’s actual pages; the map platform’s own category labels are off-limits as truth.
• Evidence or null. Every published fact carries a field-level attestation. No evidence means null, never a guess.
• Memory is self-managed. How a model handles its own context across sessions and resumes is part of what is being measured.

The only network access is a real headful Chrome the model drives itself, one page at a time, on a 5–15 second randomized throttle — no fetch APIs, no search APIs, no curl.

Buried in the corpus is a designed trap: search results falsely implying a national-brand acquisition that never happened. Across the 23 runs assessed against it — it predates the four most recent additions to the board — it fooled 12 of them, at every precision tier and context size tested. GPT-5.6 Sol and GLM-5.3 Flash declined it in all three of their replicas; Flash Q8’s best run declined it. Capacity never bought skepticism: the 1M-context frontier configurations fell for it at about the same rate as a 262k self-hosted one. This is most of what the wrong-site and false-link columns below are measuring.

Reading the board as a buyer

Best value, if you can self-host: Qwen 3.8 Flash Next at Q8 — highest floor in the field, token-frugal, and its best replica declined the trap.

One caveat before the cost numbers, because it is the thing most likely to be wrong in your situation. The open-weights runs were executed on rented GPUs, not on my desk — Flash on 2× RTX PRO 6000, the 27B on 1×. The electricity figures in the board are a model for a workstation that could hold the same weights (≈1.0 kW at the wall × BC residential Step-2 ≈C$0.15/kWh × productive hours), not a meter reading from these runs. It reproduces a separate single-GPU run I did meter directly, which is why I trust its shape — but it is still a model, not a measurement of these runs.

So there are two honest cost numbers, and which applies depends entirely on whether the hardware already exists:

The rented row is the one to argue from, because it assumes nothing. At $17–41 a run it undercuts the best frontier configuration here while being statistically indistinguishable from it on quality. The electricity row is real only if the machine is already paid for and otherwise idle — and the whole open-weights series cost $126 of rented GPU time, which is less than two Sonnet runs.

Best quality, API-only: Sonnet 5 — the ceiling of the field and fastest wall-clock by 2×, at 19–28¢ per record. Its 9.6-point spread means a bad roll lands mid-pack. GPT-5.6 Sol is the nearest challenger: top-3 quality, the second-best single run in the field, and roughly $6–9 a run on a flat subscription — with an 11-point spread of its own.

Budget disruptors: DeepSeek V4.1 Flash and GLM-5.3 Flash both land in Opus territory for two orders of magnitude less money. Their failure modes are mirror images. DeepSeek under-claims flags it didn’t browse deeply enough to verify; GLM asserts the opposite way, reading “pay your bill online” as online ordering. DeepSeek is also the fastest thing here by a wide margin — under half an hour a run, at under a quarter-cent per record.

Not recommended for this workload: the 27B. It rolled a 78.4 and a 62.6 from configs identical down to the sampler — a 15.8-point spread, the widest here, and a production lottery rather than a model choice. Its best run would have topped the board; its worst would have finished last. Opus 4.8 and 4.6 are outscored at roughly 50× the marginal cost. Opus 5 earns a place only where its 2.6-point spread — the tightest in the field — matters more than peak quality or price.

Where the errors actually land

Mean errors per run, against operator-ratified ground truth. Only one run in the entire field published a hijacked or dead domain — the ×6 category — and it was an Opus 4.6 replica. The wrong-site and false-link columns are almost entirely the acquisition trap. Territory flags are objective topology checks (a provider’s own postal code missing from its territory, implausible geography, fabricated provenance); GPT’s high count reflects the largest territory sets in the field, at 10–12.5k rows per run.

What the wider evaluation surfaced

Small context windows explode tokens and damage output. Same models, capped at 113k against 262k native: roughly 85% of a capped run’s tokens went to rebuilding discarded working memory, and quality fell with it. Capped runs took 22–24 hours and 3.1–4.6M tokens across 91–108 compactions; native runs took 3–7 hours and 0.35–0.93M tokens across 1–11. Flash gained 7.9 quality points from the window alone. The gotcha worth stealing: the harness silently assumed a 200k window for any model it didn’t recognize, so it auto-compacted at ~168k no matter what the server actually served. Nothing warned about it. Check what window your harness believes it has before you benchmark anything.

The n-gram table quantizes as a step function, which is why Q8 was worth running. Flash’s 51.2B-parameter n-gram table — a third of the model, and a pure lookup that does no matrix multiplication — stays 8-bit from Q8 down through Q5_K_XL. I checked the tensors in the shard files directly: those rungs ship byte-identical table shards, then the table halves to 4-bit below Q5. So quality is flat from Q5 up (the largest tensor is literally unchanged), and “effectively unquantized” starts two rungs below Q8. Because a lookup touches only a few KB per token, the table offloads to host RAM for free — LLAMA_ARG_OVERRIDE_TENSOR="per_layer_token_embd\.weight=CPU" moved 54 GB off the GPUs with decode unchanged and prompt processing 2.4× faster. That is what let Q8 run at full 262k native context on two cards instead of four. It also needs ≥64 GB of host RAM.

Replicas matter more than config knobs. The entire 1.75-bit to 8-bit quant ladder moved scores 0.4 points. Replica spreads ran 2.6 to 15.8. This report’s #2 and #5 were in the opposite order until the third replicas landed. Single-run comparisons measure the dice, not the models.

Agent sessions die in characteristic, preventable ways. Three runs ended without publishing, with no crash and no error — the harness reported a clean exit. Transcript autopsy found two model-side signatures. Twice, a model ended a multi-hour session by emitting a pitch-perfect imitation of the harness’s own context-compaction summary (“Let me chronologically analyze this conversation…”) and stopping: it had survived several real compactions, so that format was the most salient pattern in its context, and it pattern-completed into being the summarizer. Once, a model announced a file write and closed the turn without emitting the tool call. Both kill a session for the same structural reason — agent harnesses treat any assistant turn without a tool call as natural completion. A stop hook that checks whether the run’s required outputs exist, and injects “your outputs don’t exist yet, continue” if they don’t, turns the entire failure class into a hiccup. All three were resumed with state verified intact and are non-scoring.

What’s wrong with this eval

Everything below is a reason to discount something above. In rough order of how much it should bother you:

The judge shares a vendor with half the field. Scoring used Claude-based judging with a data-only collector fleet, and four of the nine configurations are Claude. I think the rubric’s structure limits the damage — errors are counted against a ground truth built by cross-run voting, dissent investigation and human adjudication of every disputed fact, not by a model’s free-form opinion — but I cannot rule out a vendor-shaped thumb on the scale, and I would not believe someone else’s eval with this property either. An independent judge is the single most valuable thing someone could add to this.

n=3 does not support the ranking. It supports the min-max separations listed near the top and very little else. The cost of a fourth replica across nine configurations is why there isn’t one; that is an honest constraint, not a defence.

One domain, one US state, 300 listings. Nothing here establishes that these orderings survive a different task. The workload rewards a specific blend of web verification, entity resolution and evidentiary discipline; a coding or math eval would likely order these models differently.

It is not reproducible by you. The corpus and the ~97 KB spec are anonymized because the workload is a real commercial pipeline. You can reproduce the method — sealed inputs, pre-ratified weights, n≥3, a real browser — but not the numbers.

Serving stacks differ by necessity, and sampling used each model’s shipped defaults rather than a normalized configuration. The open-weights models ran through llama.cpp; the rest through their own APIs. Pinning every model to identical sampling would be a different experiment, and arguably a less representative one, but it is a real confound.

Two configurations have unobservable token counts. GPT-5.6 Sol ran through Codex on a ChatGPT Pro subscription via an open-source protocol bridge, which reports no usage and manages context server-side, so its token and compaction columns are blank and its cost is the share of a $200/month subscription a run consumes. DeepSeek’s endpoint omits output token counts, so its volume is estimated from generated text and its cost computed from exact input/cache usage at off-peak rates.

Session seams are non-scoring. Unpublished launches were resumed with state verified intact, and wall-clock and cost figures exclude infrastructure- and evaluator-caused segments — a billing cutoff and a harness misconfiguration among them. Judging-fleet consumption is excluded from all cost figures. Reasonable people would price some of those segments into the runs.

The one thing I’d carry off this board

Not the ranking. The ranking is mostly noise at n=3, and I have tried to say so wherever it appears.

What holds up is narrower and more useful: on a long-horizon agentic workload with real browsing, real judgment calls and an evidentiary standard, a Q8 open-weights model running on two cards was not distinguishable from the best frontier API tested, had the highest floor of anything in the field, and cost $17–41 a run to rent against $56–85 to call.

The lever that made that true was not the weights. The same model family scored 7.9 points lower with its context window capped, and the quant ladder from 1.75-bit to 8-bit moved scores by 0.4 — forty times less than the spread between identical replicas of the same config. Window, harness and replication count dominated everything I could change about the model itself. If you are benchmarking local models against APIs and getting results you don’t like, I would check those three before I’d check the weights.