Qwen 3.8 Flash Next on vLLM not natively compatible with Anthropic API

Qwen 3.8 Flash-Next in FP8 on vLLM scored 64.3 on an agentic eval that the same model in Q8 on llama.cpp scored 75.7 on. Identical task, identical prompt, identical Claude Code binary. Its per-step reasoning was about half as long.

Effort settings, sampler defaults, request shape, chat template, engine numerics and weight precision were each tested and cleared. The cause was in the transport. vLLM’s native Anthropic Messages bridge hands the model its own earlier thinking back on every turn. The model imitates it, the imitation joins the history, and within a few hundred steps 40% of its “reasoning” is one literal sentence:

I need to investigate this further. Let me check the details.

That string is 62 characters long. Running the same FP8 server through a litellm proxy to vLLM’s OpenAI endpoint — which never delivers prior thinking, by accident — restored full-length reasoning immediately, held it through seventeen compactions, and scored 74.0, inside the Q8 band.

The setting

The eval is a 300-row enrichment pipeline driven end to end by Claude Code with a real browser, scored against a rubric ratified before any results existed. Every backend is driven by the same client, the same prompt, the same tools. The workload’s business domain is anonymized here as everywhere on this site; the mechanism below has nothing to do with it.

• The anchors: Flash-Next Q8_0 GGUF on llama.cpp, driven through a litellm proxy that speaks Anthropic Messages on one side and OpenAI chat-completions on the other. Three runs: 75.8 / 73.0 / 78.4, mean 75.7.
• The new rung: the official FP8 checkpoint on vLLM 0.29, two CMP 170HX plus an RTX 3090, pipeline-parallel 3, driven through vLLM’s native Anthropic Messages bridge (/v1/messages, no proxy). Three runs: 55.8 / 70.0 / 67.2, mean 64.3. About 105 tok/s decode with multi-token prediction and roughly C$0.40 of electricity a run, which is exactly why the gap was worth chasing.

The judges’ verdict on the FP8 trio was that precision was not where the points went. No fabricated values, no leaked hijacked domains. The errors were behavioural: a known acquisition trap taken 3 of 3 times, shallow reads, under-claimed flags. That is the profile of a model not thinking hard enough, so the thinking became the question.

The symptom, measured three ways

Claude Code’s transcript keeps every assistant message with its thinking block, so per-step reasoning length is directly measurable.

Every run through the bridge thinks about half as much per step as every run through litellm, regardless of quantization — and the same FP8 server through litellm sits with the Q8 anchors. Thinking length alone is not a score predictor: two llama.cpp runs at Q4 and Q6 sit in between. The shape is.

Bucketed against context length, bridge thinking decays: run 36b goes 239 → 159 → 119 → 62 characters as context grows past 190k. The litellm runs, bucketed by message order, don’t (23c by quartile: 290 → 460 → 348 → 413).

The cut that gave the mechanism away is within a single compaction segment. Claude Code compacts every ~190k tokens; each compaction wipes the history and starts fresh.

A floor with the same value, 62, in run after run is not statistics. It is a string.

By mid-run the bridge model emits that sentence on a quarter of all steps, and it also creeps in as a prefix to steps that do carry real reasoning. Ten consecutive steps from the middle of 36b, with the workload’s identifiers removed:

900: THINK(1204) 'thin service list: installation and servicing of …'
       -> Bash: python3 runs/_scratch/radius_zips.py …
901: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> Bash: python3 <<'EOF' … zips=sorted(json.load(…
902: THINK(  54) '26/60. Unit 27. Read the input.'
       -> Bash: python3 -c "… d=json.load(open('runs/…/04_enrich/inputs/…
903: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> navigate_page: [operator site]
904: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> take_snapshot
905: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> Bash: grep -nE "link \"|heading|StaticText" …/snap4_home.txt
906: THINK( 651) 'I need to investigate this further. Let me check the details.  [operator]: nationwide commercial delivery…'
       -> Bash: grep …
907: THINK( 334) 'I need to investigate this further. Let me check the details.  Homepage: 24/7 emergency response…'
       -> navigate_page: [operator site]/delivery/
908: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> take_snapshot
909: THINK(  62) 'I need to investigate this further. Let me check the details.'
       -> Bash: grep -nE "heading|StaticText" …/snap4_delivery.txt

The same stretch of work on the Q8 anchor has no step under 56 characters and five over 500, including a 3,913-character deliberation about which of two datasets to trust and a step beginning “I hand-wrote the list, but that was wrong — I fabricated/estimated it instead of using the computed set.” That self-correction is what the bridge runs stopped producing.

Where the reasoning goes missing is specific. Median thinking on turns whose first action is a given tool:

The bridge runs still think before navigating. They stop thinking on the steps that look at what came back and decide — snapshot and grep turns, where 291 of 36d’s 327 fillers and 257 of 36b’s 289 land. Those are exactly the steps where this pipeline’s classification and flag judgments get made.

What it was not

Each of these was a live suspect and was tested directly.

The single remaining difference was the transport, and the within-segment decay pointed at something that accumulates with history and resets with it.

The mechanism

Claude Code sends every prior assistant turn back verbatim, thinking blocks included, on every request. That is required by the Anthropic API for tool-use continuity — the blocks are signed and the client cannot drop or edit them. What reaches the model depends on what sits in between.

vLLM’s Anthropic bridge converts each assistant message’s thinking blocks into a reasoning field:

elif block.type == "thinking" and block.thinking is not None:
    reasoning_parts.append(block.thinking)
...
if reasoning_parts:
    openai_msg["reasoning"] = "".join(reasoning_parts)

Both chat templates in play read that field and render it back into the prompt for every history turn after the last user query:

{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
    {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}

In a Claude Code tool loop the “last user query” is the task prompt — tool results come back as user messages beginning <tool_response>, which the template does not count as queries. So every turn since the task prompt, or since the last compaction summary, is “after the last query”, and every turn’s thinking is rendered. Setting preserve_thinking=false does not change that.

litellm converts the same blocks into a thinking_blocks field instead. Neither llama.cpp nor vLLM’s chat-completions endpoint reads thinking_blocks, and the template reads only reasoning_content, so the history think blocks render empty:

<|im_start|>assistant
<think>

</think>

<tool_call>

A parity harness takes one real request, converts it both ways using litellm’s actual adapter on one side and a line-for-line mirror of the bridge on the other, renders both through the served template, and diffs them pattern by pattern. For a 15-message slice with 7 assistant turns: system prompt identical, effort instruction identical and at the same offset, tools block identical, history turn count identical, tool responses identical, generation prompt tail identical, sampler params identical. One line differs:

The unified diff of the two rendered prompts is nothing but those six filled-in think blocks.

Why carried thinking becomes a loop. The model sees, on every step, a history in which its own previous thoughts were short. It produces another short one. That one joins the history. The distribution tightens toward the shortest acceptable output, which for this model under this template is a single stock sentence. Compaction wipes the history, the model briefly thinks at length again — 36b’s fresh segments open at 305, 411, 222 — and the decay restarts. Synthetic probes reproduce the direction: with 12 prior tool turns whose thinks are one short line each, reasoning on a check question drops from ~1,400 to ~370 characters; with 12 substantive prior thinks it stays at ~3,900.

Same server, the other transport

Run 36e drives the identical FP8 vLLM server — stock template, no restart — through litellm and vLLM’s OpenAI endpoint, the anchors’ client path. The launch gate rendered a litellm-shaped history turn through the live server first and confirmed the history think block came back empty.

Seventeen compactions and no drift toward a floor. It matched the Q8 anchor from the first window onward: at 160 messages 283 against 284, at 560 messages 338 against 337, at the end 342 against 361.

Score: 74.0, inside the Q8 band of 73.0–78.4, above one of the three anchors and 1.7 under their mean. Same weights, same server, same template, same effort, same sampler. Only the history policy changed. The judged findings, against the four bridge runs:

• The designed acquisition trap was declined, the first time in five FP8 runs. Every bridge run had taken it and invented a parent company, 6 points each time.
• Zero wrong sites, zero wrong links, no leaks. Two entities that bridge runs had resolved to a corporate parent’s domain or a stale acquirer’s post came out correct or correctly blank.
• Every record 36d dropped was published — twelve in total, the most in the family.
• Fabrication sweep clean at three times the reasoning: 30 of 33 founding years match the cross-run majorities, and the two 36d had invented came out correct. Fewest under-claimed flags in the family, 8 against 19–25.
• Where the points went: blank chain links, own-zip territory misses, junk rows and two flag misreads. That is the Q8 anchors’ deduction profile, not the bridge family’s.

Restored reasoning is not free: 6.29 hours against 2.1 for 36d, about C$0.94 of electricity, 619 page navigations, and 1.8M output tokens of which 1.14M were reasoning. Claude Code compacted 17 times because through a proxy it compacts on its own token estimate rather than server counts.

Is this a Claude Code bug? No. A bridge bug? Not exactly

Claude Code behaves the same way against Anthropic’s own backend — it sends thinking blocks back because the API requires it. Anthropic’s servers then strip thinking from earlier turns but keep the current turn’s, and with interleaved thinking Claude reads its own prior reasoning between tool calls. In a pipeline run the stretch between two compactions is one long turn, so Claude sees all of its loop thinking. The bridge gives Qwen the same regime. As an emulation of the Anthropic API, the bridge is faithful.

The difference is the model. Claude is trained to reason on top of its own carried thinking. Flash-Next, under Qwen’s templates, imitates it and degenerates. Qwen’s own multi-turn guidance says to exclude historical thinking from context — and the Q8 anchors, which scored best, ran exactly that way, by accident of the litellm adapter.

So the history policy has to be model-specific: Anthropic’s policy for Claude, an empty history for this Qwen model. That policy belongs on the server, next to the model it serves — not in the client, which must keep sending the blocks whatever the backend.

Fixes

Three ways to get the empty-history rendering on vLLM, all server-side or proxy-side:

1. Proxy path, validated live by 36e: drive vLLM’s OpenAI endpoint through litellm. No server change. Costs a proxy hop, zero usage counts in the client’s transcript, and compaction on the client’s own estimate rather than server counts.
2. Bridge patch: stop appending assistant thinking blocks to reasoning_parts — treat them like redacted_thinking — ideally behind a per-model option. Upstreamable.
3. Template edit: never set reasoning_content from history turns. One line, but it changes the model’s contract for every client of that server.

Proof for any of them is mechanical: run the parity harness until every pattern reads SAME and the diff is empty, then one replica through the fixed path must show 36e’s signature — no filler sentence, flat segments — rather than 36d’s.

What changed in the numbers

One run is not three. But the gap moved from 10.6 points to 1.7 with a single change of history policy, on a run that cost under a dollar to serve. Two more replicas will settle whether 74.0 is the rung or a good day, and a patched bridge gets its own run.

If you are serving an open-weights model to an agentic client through an Anthropic-compatible endpoint, this is worth ten minutes: dump one rendered prompt and look at whether the history <think> blocks are full or empty. The failure is silent, it looks like a bad model, and every conventional suspect tests clean.