#local-llm
14 posts tagged local-llm. · All tags
- The Prompt Did Not Do It, Splitting the Stages Did I tried to fix thin summaries with the prompt and made them worse. Separating number extraction from sentence writing raised recall 18 points and presence 2.6 points together
- I Cut the Input to a Third, and What I Thought I Had Fixed Split in Two Two-stage processing fixed both numeric accuracy and omission. Measuring again on short inputs showed one was a length problem and the other was a structure problem
- I Measured Four Models on 200 Documents, and What I Did Not Measure Overturned the Result On numeric fidelity the challenger won. Then I found that Korean number spacing was broken in all 200 outputs
- Measured Separately Is Not Measured Together, and 3% Utilization Measured thinking mode on an offline set, concluded no effect, then it reversed on a multi-step set. Meanwhile a never-measured combination went into production. The max_tokens cap was a breaker, not a budget, and after four months of tuning, the hardware's utilization was 3%
- The Grader Was Wrong More Often Than the Model Over 800 validation runs while I slept; the morning failure list had 8+ items that were measurement-tool defects. Fixed the tools, then 1,000 config experiments on the real defects that remained. Three cleared, four walls
- I Deleted the Gateway and Fixed the Wiring The gateway I set up two days earlier was attaching stock-market open days even to cooking questions. I repeated the exact pattern I was stopped from in April, and once it was gone, most of the chat quality problems were silently ignored settings, not the model
- I Decided to Re-measure the Hardware from Scratch An orphan number called sm_121, a 6:0 SGLang win that flipped once speculative decoding was on, and a public-benchmark leader losing on my own labeled data. A day re-measuring the DGX Spark at three layers: chip, engine, model
- One Division Eliminated Half the Model Candidates Measured DGX Spark memory bandwidth at 231GB/s. Divide by active parameters and you get the decode ceiling; every dense model was eliminated by arithmetic alone
- I Trusted the Community and Halved My Decode Speed Believing "speculative decoding and multimodal are incompatible," I turned off MTP and 42.8t/s dropped to 19.4t/s. Reading the source, the code only warns and proceeds, and meanwhile another trap had been armed
- Settings That Are Bad Alone and Good Together Shrinking chunked prefill alone made tail latency worse; combined with priority scheduling, worst-case TTFT dropped from 51.8s to 2.0s. A night of rejecting four community recommendations by measurement
- I Lost 2.5 Months of Logs in My First Hour Back In the first DGX Spark session in two months, "check the logs" turned into a container recreate and 334MB vanished. What survived was 14 days that another system happened to be watching
- Shrinking the Prompt Made It 55% Faster, and Lost What Actually Mattered The compression that produced −55% was throwing away the core of the domain. A story about a metric I wasn't measuring collapsing.
- Five Days of Fine-Tuning Lost to a 108-Character Prompt I spent five days on LoRA for Gemma 4 26B-A4B and it barely differed from no tuning at all. What happened when I attacked the same task with prompt compression instead.
- I Was Running a Model per Project, So I Merged Them into One LLM Server Standing up a vLLM server on a single DGX Spark, splitting it out instead of running a model per project, and wiring up monitoring on day one