I Decided to Re-measure the Hardware from Scratch
An orphan number called sm_121, a 6:0 SGLang win that flipped once speculative decoding was on, and a public-benchmark leader losing on my own labeled data. A day re-measuring the DGX Spark at three layers: chip, engine, model
Four days of tuning on the DGX Spark, changing settings only. But I had never doubted the engine and model chosen back in April, and the point came up that the tests weren’t rigorous enough to tell models apart. So I re-measured three layers in order: chip, engine, model.
What I set out to do
Four days of tuning by changing settings. But I had never once doubted the engine and model themselves. I left the vLLM and Gemma 4 26B-A4B I chose in April as they were and only worked on top of them.
Shouldn’t the tests be more rigorous before we can tell the difference between models?
That day I re-measured three layers in order: chip, engine, model.
How I asked
Chip first. The GB10 is sm_121. B200 is sm_100, the RTX 50 series is sm_120, and their kernels are not mutually compatible. No upstream project has GB10 CI. The GB10 support PR for vLLM had gone unmerged for five months, and the reasons were CI cost and market size. TensorRT-LLM failed on both release candidates, but digging in, it wasn’t a GB10 problem but a combination problem with Gemma 4. Gemma never appears once in NVIDIA’s playbook. The practical options are two: vLLM and SGLang. That means we have to play the role of CI, and that’s why the validation scripts had grown to 59.
Engine. I compared vLLM and SGLang under the same conditions (both without speculative decoding).
Where it broke
SGLang won all six items. Single 23.5t/s vs 19.4t/s, first token 0.09s vs 0.18s, concurrent 1, 2, 4, 8, 16 all of them. By the table alone there was ample reason to switch.
But that wasn’t the configuration we’d ship. I turned on speculative decoding for both and re-measured.
| vLLM + MTP | SGLang + MTP | |
|---|---|---|
| Single decode | 42.8t/s | 40.0t/s |
| Disclosure throughput | 1,323/h | 1,072/h |
| Figure-in-source rate (share of output numbers that actually appear in the source text) | 98.0% | 98.0% |
| Agent bench | 24/24 | 24/24 |
Quality was completely identical and only throughput flipped, by 19%. SGLang had a side trap too. It returns 200 for a nonexistent model name. vLLM gives 404. Typos get swallowed silently. I decided to stay on vLLM and upgraded from 0.22.1rc to v0.27.1. Disclosure throughput rose 31% to 1,733/h.
How I fixed it
Model. I measured the two surviving MoE candidates with the same harness. The criterion wasn’t speed but numerical fidelity against the 13,206 labels accumulated in the disclosure-summary service.
| Gemma 4 26B-A4B | Qwen3.5-122B-A10B | Qwen3.6-35B-A3B | |
|---|---|---|---|
| Single decode | 39.4t/s | 13.9t/s | 41.0t/s |
| Disclosure throughput | 1,733/h | 628/h | 1,915/h |
| Figure-in-source rate | 97.9% | 86.8% | 90.3% |
| sentiment agreement | 96% | 96% | 83% |
Qwen3.6-35B-A3B dominates Gemma on public benchmarks. MCPMark 37 vs 14, OmniDocBench 89.9 vs 74.4. But on our data, 10% of the numbers it outputs aren’t in the source. Turn on thinking mode and it becomes accurate, but at 62.5 seconds per document, four times Gemma. Fast and inaccurate, or accurate and four times slower.
The 122B’s quality edge was real. In risk review it caught a mismatch between position size and stop-loss width: “a 3% stop on a 30% position has a high probability of early liquidation.” Gemma gave generalities. But 13% of its output figures weren’t in the source and throughput was one third.
The same direction came out twice. Models strong at general reasoning actually fall behind on grounded extraction. The axis public benchmarks measure and the axis I’ll use are different. Confirmed: keep Gemma 4 26B-A4B.
Lessons
- Measure benchmark tables in the configuration you’ll ship. A 6:0 table without speculative decoding flipped once it was on. Default-state comparison is reference only.
- Model replacement is decided by my labeled data. The public-benchmark leader lost in our own evaluation. The same pattern repeated across two models.
- Numerical fidelity, not speed, was the deciding factor. In document summarization, 10% of numbers absent from the source isn’t offset by an 11% throughput gain.
- If the chip has no upstream CI, I am the CI. The pile of sm_121-specific validation scripts is not a cost, it’s a condition.
- Beware APIs that succeed silently. An engine that gives 200 for a nonexistent model tells you about the typo days later.