The Harness Beat the Model

I’ve been running a bakeoff: open-weights models against frontier Claude models on a real production workload — a data-enrichment pipeline that runs unattended for hours, makes hundreds of judgment calls, and gets scored against human-adjudicated ground truth. (The business domain is anonymized throughout this site; the numbers are not.)

The first result worth publishing wasn’t about any model. It was this: the same 27B model finished last under one harness and set the quality record under another.

Same weights, in two serving formats. Same sealed inputs. Under my first agent stack the model limped to the bottom of the table; served through a different runtime and driven by a different agent loop, it posted the strongest single result of the round — winning more per-item comparisons than anything else I tested, frontier models included.

Where the delta actually lives

Unpacking that gap took a while, because none of its parts show up on a spec sheet:

Chat template. The served template silently determines whether reasoning effort does anything at all. One template family rejects out-of-range effort strings with an HTTP 500; a popular community fix coerces them instead. Neither behavior is documented. If your harness thinks it requested high effort and the template disagrees, you benchmarked the wrong configuration and nothing warned you.

Effort routing, verified — not assumed. I ended up putting a proxy in front of every open-weights model that pins reasoning effort server-side and drops whatever the client sends. Each run starts with a differential proof: a probe that must succeed at the pinned effort and a sibling that must fail (or stay silent) at a decoy effort. If the differential doesn’t hold, the run doesn’t start. This one gate has caught more silent misconfigurations than any other check in the project.

Resume mechanics. Long agentic runs die and resume. Resuming by replaying a near-context-limit transcript drives the session straight back into the wall it just hit; resuming as a fresh session against a progress file on disk works. The difference is worth more than a quantization tier.

Token accounting. One agent framework undercounted replayed thinking tokens — real usage ran 20–60% above the counter depending on phase — which meant its believed context window was fiction, which meant sessions ran into a wall the framework didn’t know was there. The model took the blame in the scores until I found the counter.

What I do differently now

Every comparison on this site holds the harness fixed and verified before it varies the model. Launch gates prove effort routing, vision passthrough, and browser access independently — a run that can’t prove its configuration doesn’t get to produce a score.

The uncomfortable takeaway generalizes: when you read that model A beat model B inside an agent product, you’re mostly reading about the product. The weights are load-bearing. The stack decides what they can carry.