The Local LLM That Read "6 Shares" as 0.06
Moving the trading bot's daily LLM roles to a self-hosted model, and pinning down the misreading bug with a byte-exact DB reproduction and A/B measurements
The trading bot’s daily judgment roles run through headless Claude, and to cut costs I’m moving some of them to a local gemma server on the DGX Spark. This was the day to fire the strategist prompt, the one that proposes target weights, at the local model as-is and look at the quality. On the first real run the model copied holding quantities straight into the weight slots, and some of those walked right through validation.
What I set out to do
The trading bot has roles that run daily through headless claude -p. Market analysis, strategist (proposing target portfolio weights), cross-examination, self-diagnosis. To cut costs I’m moving some of these to a vLLM-based gemma server on the DGX Spark, and today was the day to fire the strategist prompt at the local model as-is and check the quality.
How I asked
I pasted the local LLM server integration guide into Claude Code and had it test with the prompt used in production. The first real run produced an accident. In the holdings notation - 069500 (KODEX 200): 6주, 평단 …, gemma copied the 6주 (6 shares) straight into a target weight of 0.06, and 22주 (22 shares) into 0.22. 8 out of 15 runs.
The scary part isn’t the failures. If the sum of the copied weights exceeds 1.0, validation (RANGE_FAIL) catches it and discards it, but if the sum happens to be 1.0 or under, it passes straight through. In fact, the Aug 14 prompt copied only two tickers, summing to exactly 1.00 — it walked right past the defense line.
Where it broke
It wouldn’t reproduce. On the server side, I hand-reconstructed the prompt from the report and ran it 24 times: 0 hits. That meant the reconstruction differed from the real thing, so I had it redo with the original and nailed down the conditions.
Dump the DB value exactly as-is. Do not retype or tidy it by hand — we don’t yet know whether whitespace, line breaks, or section order is the cause.
Claude Code pulled the prompt byte-for-byte from the production DB, checked its md5 hash against the file, and ran it: original 30/30, confirmed reproduction. Reconstruction 0/24 versus original 30/30 — even when the content looks the same, only the exact original counts for reproduction. JSON parse failures got mixed in midway and confused things briefly, but that was a separate issue: the harness’s max_tokens of 900 was too small and truncated responses. Claude checked finish_reason, separated it from the bug, and spun up a separate auxiliary run at 1400.
How I fixed it
I narrowed the cause with 4 notations (original / colon removed / quantity removed / warning label) × 15 runs A/B. Removing the quantity makes it vanish cleanly at 0/30. Removing just the colon brought large-quantity copying to 0, but residual small-quantity cases were ambiguous, and changing the quantity from 56 to 41 shares made the weight follow to 0.43 — the quantity notation is the cause.
Two fixes. Remove quantities from the holdings notation + remove the colon after the ticker code, and change theses in the output schema from a nested map to an array (nested maps broke parsing more than half the time from whitespace blowup in structured output; arrays were 13/13). Re-verification used prompts freshly rendered by the fixed generator, not the DB original: 2 prompts × 2 conditions × 15 runs = 60/60 passed. This input/output contract went into a decision document, and tests lock the prompt example and schema constants together. I also nailed down one thing about the migration approach.
Don’t swap out the existing Claude track — add a new track. If you swap it, the A/B becomes a lie.
Lessons
- Swapping LLMs doesn’t work on “same prompt, should be fine.” A single notation (
6주) flipped the output. - Reproduce with the original bytes. The hand-reconstructed prompt was 0/24; the DB dump original was 30/30.
- The most dangerous failure isn’t the one validation catches but the one that passes by luck. A sum of 1.00 was a stable state at 15/15.
- Side issues like response truncation must be separated from the main bug and counted apart, or the A/B gets muddied.
- Don’t touch the control group in a comparison experiment. Migrate on a new track.