I Was Running a Model per Project, So I Merged Them into One LLM Server
Standing up a vLLM server on a single DGX Spark, splitting it out instead of running a model per project, and wiring up monitoring on day one
Two things now needed an LLM: the disclosure-summary batch, and a watchdog batch running once an hour on the Mac mini. Instead of a model per project, I stood up a single vLLM server on the one DGX Spark at home and made a separate project that does nothing but serve the model. Splitting it out left a new chore: seeing the server’s state from outside.
What I set out to do
I have an NVIDIA DGX Spark (GB10) at home. 128 GB of unified memory, with the CPU and GPU sharing that memory.
Two things now needed an LLM: the batch job for the disclosure-summary service, and a watchdog batch that runs once an hour on the Mac mini. Having each run its own model is a waste.
“I’m trying to run the same model. Would one be enough? It’ll be used by 2 different projects.”
One was enough, it said. So I made a separate project that does nothing but serve the model. vLLM is a serving engine that holds model weights in memory and takes inference requests over HTTP. Run one of those, expose it on port 8000 behind an OpenAI-compatible API, and any caller already using the OpenAI SDK only has to change the address. Each service brings its own prompt for the domain logic, and the server only serves the model.
I wondered whether creating yet another project was the right call, but flipping it around made it obvious. If I had to touch both application repos every time I swapped the model or tweaked engine settings, that would be the strange thing.
How I asked
It was done in three commits in one day. vLLM serving and monitoring at 1:56 p.m., the embedding service (multilingual-e5-large, on port 8001 next to the main server) at 2:05, and security cleanup and log collection at 3:07.
I put the embedding service up alongside it for when I add a search feature later. Nothing uses it right now, but since it would go on the same machine anyway, I set it up in advance. Whether setting up something nobody uses yet is a good call, I don’t know.
Monitoring went in together from the start. If you split the server out, you need a way to tell which side a problem is on when one happens. That’s the price of the split. I asked whether vLLM’s status and logs could be viewed from outside, and put Prometheus and Grafana on top. Logs are collected separately.
Where it broke
Wiring it up didn’t mean it showed anything. Login worked, but there wasn’t a single dashboard, and when I made a test call to the embedding service, nothing showed up in the graphs.
“The embedding test call I just made isn’t showing up on the Grafana dashboard. Are metrics actually being collected? Is this another thing I have to do separately?”
It was. Attaching an exporter, registering it as a scrape target, and building a dashboard are each separate tasks. You need all three before any numbers appear.
How I fixed it
Only after I built the dashboard myself did the metrics start showing. The current serving config looks like this.
--model nvidia/Gemma-4-31B-IT-NVFP4
--quantization modelopt
--max-model-len 8192
--gpu-memory-utilization 0.85I chose the NVFP4 quantized build. It is a format that packs the weights into 4 bits, and --quantization modelopt names the backend that reads that format. Since the unified memory is shared with the CPU, I figured smaller weights would be better, and it does load fine.
I left gpu-memory-utilization at 0.85. Normally that’s a conservative value.
Remaining problems
It’s slow. About 7 tokens per second on a single stream. Part of me thinks that’s just what a 31B model does, but with NVFP4 shrinking it to 4-bit, this doesn’t seem right. Looking at GPU utilization, there’s a fair amount of idle time. It doesn’t look compute-bound; it seems to be stuck somewhere else, but I don’t know where yet.
(This gets answered in episode 27. What was blocking it was not compute but memory bandwidth, and once I knew that, a single division filtered out my model candidates.)
A context of 8192 is also short. Disclosure texts over 10,000 characters are common, and I’m not sure this will cover them. Korean has fewer characters per token, so 10,000 characters can exceed 8,000 tokens. If it gets truncated, the tail is dropped wholesale, and in disclosures the important figures often come at the end.
One more minor thing. While naming the working directories for local and remote, one ended up singular and the other plural. The distinction started from a typo, so I even left a note to not get confused. It’s nothing now, but I’ve been bitten a few times by things like this later on.
(It did bite. Five months later I found this server’s systemd unit had been failing to restart over 50,000 times, because the s was missing from the working directory path. The container was already up, so serving was unaffected, which is why nobody noticed. A reboot would have been how I found out.)
Lessons
- If you split a service, split the observability with it. If you can’t tell “whose fault is it,” the split isn’t worth anything. Wiring up monitoring on day one wasn’t a feature; it was the cost of the split.
- Having monitoring wired up and having metrics visible are two different things. Exporter, scrape, and dashboard are each separate.
- Once the model server is outside the app, each service only needs its system prompt and a standard API call. Whether I can keep holding this boundary seems to be the real question.