The Grader Was Wrong More Often Than the Model
Over 800 validation runs while I slept; the morning failure list had 8+ items that were measurement-tool defects. Fixed the tools, then 1,000 config experiments on the real defects that remained. Three cleared, four walls
The GPU box at home runs vLLM and serves a local model directly. With the engine and the model picked, this was the first proper measurement of what the server could do, so I fixed the scenarios and the pass thresholds up front and let it run all night while I slept. The question in the morning was whether the failure list was really the model failing.
What I set out to do
With the engine and model settled, it was time to measure what this server can actually do. Tool calling, long-document comprehension, domain knowledge, structured output, search. On the night of Aug 19 I drew up 40 scenarios and 4 conditions, and nailed the pass thresholds down in a document before looking at any results.
I’m going to sleep now and I’ll come check tomorrow morning. Test it thoroughly, to the point of using up all the tokens.
How I asked
It ran over 800 times overnight. The day’s total came to some 1,700 runs plus about 400 search experiments on top. In the morning I opened the failure list.
Where it broke
A large share of what was marked as failure was a grader problem, not a model problem.
| Model’s answer | What the grader looked for | Actual |
|---|---|---|
| “09:00 ~ 15:30” | Only the 9시 pattern accepted | Correct |
| “9조 2,000억” | Only 9.2조 accepted | Correct |
| “1조 2,340억원” | Number extraction yielded “1” | Correct |
| “첫 번째 금요일” | 첫째 금요일 | Correct |
The domain-knowledge item went from 84/93 to 93/93. Some of the harness itself was broken too. The long-document needle-in-a-haystack test was written to insert the needle at a newline boundary, but the disclosure source text had no newlines. From depth 20% to 80%, every needle landed at the same spot, character 1,877. The entire depth axis was fake. Fixed, 40/45 was actually 74/75. The calculator tool fixture returned 46.6 regardless of input, so even when the model delegated correctly it failed grading. The 5 empty answers weren’t wrong answers but insufficient max_tokens. Raised it and 71% became 97%.
False failures exceeded 8. Once it went the other way. While loosening the grading I nearly passed “operating profit 499 trillion,” but re-reading the source, it was a faithful copy of an actual article. Not a hallucination but a hedging failure. I split it into hallucination 0/9 and hedging failure 2/3 and recounted. The one rule set that day: an item marked as failure is not a failure until the source has been read.
How I fixed it
After fixing the tools, the real defects that remained were these. Beyond 20 tools, raw tokens leak into the answer. The boundary is between 18 and 20, and it’s a cliff, not a slope. tool_choice is completely inert because the template doesn’t reference it. Using json_schema and tools together corrupts figures only when combined with certain prompt phrasings. Fall into a tool loop and you get an empty answer.
To see whether these could be cleared by configuration, I ran 7 kinds of experiments, about 1,000 runs.
Three cleared. Cutting the built-in tools from 28 to 15 reduced per-request overhead from 4,251 tokens to 1,994 and the leaking disappeared. Two lines of system prompt took flagging of common-sense departures from 0/5 to 4/5. Receiving numeric fields as string gave 0/10 figure corruption.
Four walls. Search fell short in all 8 conditions, and the cause wasn’t configuration. Financial sites block bots and return 3 bytes. It took 400 runs before I admitted it. Tool loops went from 6 cases to 8 when I added a penalty. Long-document and conversation isolation fell short in all 3 conditions. tool_choice: nothing to do server-side. Sampling was irrelevant. However I shook temperature and top_p, stable items were 120/120, and the noise floor on repeated measurement was ±8%p.
Claude Code’s conclusion was exact. “The four walls are the calling side’s code, not server settings.”
Lessons
- On seeing a failure, suspect the tool before the model. More than half of one night’s failure list was regex, fixture, and harness defects.
- An empty answer is a measurement failure, not a wrong answer. Use finish_reason to separate truncation from wrong answers.
- Write the pass threshold before looking at results. Set it afterward and the line moves every time the grader gets fixed.
- Beware overcorrection. Loosening the grading also gets verified against the source. Hallucination and hedging failure are separate items.
- Measure the noise floor first. A difference within ±8%p is not an improvement. Half of the 7 experiments were inside this floor.