Running the Eval on My Own Silicon

Everything on this site pointed here: could the eval that frontier APIs and rented datacenter GPUs had been running finish on the desk — same sealed inputs, same browser, same native context, same scoring?

Short version: yes, in ~4.5 hours — faster than two of the three rented BF16 runs of the same model — for about C$0.68 of electricity. The long version contains the two best systems findings of the whole project.

The +17% environment variable

Bringing up the second GPU was its own comedy (my build had kernels for only one architecture; the new card died instantly with no kernel image is available until I rebuilt for both), but the payoff hid in a week-old llama.cpp commit: multi-GPU had been silently excluded from graph-level stream concurrency by an over-broad guard, plus a genuine bug where a sync event could be created on the wrong device. With the fix in and the flag on:

GGML_CUDA_GRAPH_OPT=1	decode	prefill	power
off	54.1 tok/s	1,548 tok/s	279/122 W
on	63.1 tok/s	1,542 tok/s	278/123 W

Seventeen percent more decode at identical power, because batch-1 inference is a parade of kernels too small to fill a GPU, and concurrent streams let independent ones overlap. Prefill didn’t move — big batches never had the bubble problem. Free performance, one environment variable, on any multi-GPU llama.cpp box.

(Two adjacent bring-up findings, briefly: speculative decoding’s draft context proved brittle to tensor-split ratios — only a near-even split would load at all, so “shift layers to the faster card” lost to “keep MTP alive” — and the default split ignores that your display card is already spending ~800 MB on being a display.)

The wall that stopped the bigger model

I first tried a larger MoE via expert-cache streaming — experts resident in system RAM, a GPU cache serving the hot set. Decode was fine. Prefill ran 3 tok/s cold, ~16 warm — and an agentic harness front-loads a ~15k-token system prompt before saying hello. That prices a session handshake at fifteen minutes to an hour, and a post-compaction context rebuild at four to twenty. Infeasible — not slow, infeasible.

The arithmetic is brutal and general: prompt processing sweeps every expert per batch, so 77 GB of host-resident tensors against 32 GB of RAM means NVMe-fed prefill forever. Cache tricks accelerate decode’s sparse access pattern; nothing accelerates a dense sweep you can’t hold in memory. For agentic workloads, system RAM is the binding constraint on expert offload — the upgrade that matters on my box isn’t a third GPU, it’s going to 128 GB.

The run itself

So the capstone ran the model that fits: a 27B at the largest quant the pair can hold (the ~94% top-1 agreement tier), both cards, full native 262k context, F16 KV, speculative decoding on. Against the rented datacenter runs of the same model at the same window:

	rented (1× datacenter GPU, n=3)	local (4090+3090)
wall clock	4.3–6.9 h	~4.5 h
output tokens	567–928k	288k
records published	62–63	64
marginal cost	$11.78–19.03	≈C$0.68

One asymmetry disclosed: the rented rows served BF16 weights; the local run serves the Q6-class quant. The measured quant ladder on this pipeline spans 0.4 points, but they are scored as separate configurations on the board.

The token column is my favorite: fastest thinking, not just fastest serving — fewer compactions and a warm, stable session meant less rework. The thermal soak ran concurrently and passed; the quality score lands on the Gauntlet board alongside everything else.

The through-line of the whole project, closing where it started: the model was never the hard part. The window, the harness, the kernels, the RAM, the junction temperature — the system is the product, and it’s finally one I’d deploy.