#vllm
10 posts tagged vllm. · All tags
- Measured Separately Is Not Measured Together, and 3% Utilization Measured thinking mode on an offline set, concluded no effect, then it reversed on a multi-step set. Meanwhile a never-measured combination went into production. The max_tokens cap was a breaker, not a budget, and after four months of tuning, the hardware's utilization was 3%
- The Grader Was Wrong More Often Than the Model Over 800 validation runs while I slept; the morning failure list had 8+ items that were measurement-tool defects. Fixed the tools, then 1,000 config experiments on the real defects that remained. Three cleared, four walls
- I Deleted the Gateway and Fixed the Wiring The gateway I set up two days earlier was attaching stock-market open days even to cooking questions. I repeated the exact pattern I was stopped from in April, and once it was gone, most of the chat quality problems were silently ignored settings, not the model
- I Decided to Re-measure the Hardware from Scratch An orphan number called sm_121, a 6:0 SGLang win that flipped once speculative decoding was on, and a public-benchmark leader losing on my own labeled data. A day re-measuring the DGX Spark at three layers: chip, engine, model
- One Division Eliminated Half the Model Candidates Measured DGX Spark memory bandwidth at 231GB/s. Divide by active parameters and you get the decode ceiling; every dense model was eliminated by arithmetic alone
- I Trusted the Community and Halved My Decode Speed Believing "speculative decoding and multimodal are incompatible," I turned off MTP and 42.8t/s dropped to 19.4t/s. Reading the source, the code only warns and proceeds, and meanwhile another trap had been armed
- The Local LLM That Read "6 Shares" as 0.06 Moving the trading bot's daily LLM roles to a self-hosted model, and pinning down the misreading bug with a byte-exact DB reproduction and A/B measurements
- Settings That Are Bad Alone and Good Together Shrinking chunked prefill alone made tail latency worse; combined with priority scheduling, worst-case TTFT dropped from 51.8s to 2.0s. A night of rejecting four community recommendations by measurement
- I Lost 2.5 Months of Logs in My First Hour Back In the first DGX Spark session in two months, "check the logs" turned into a container recreate and 334MB vanished. What survived was 14 days that another system happened to be watching
- I Was Running a Model per Project, So I Merged Them into One LLM Server Standing up a vLLM server on a single DGX Spark, splitting it out instead of running a model per project, and wiring up monitoring on day one