I Got Stopped From Building One More Thing in Front of the Server
I cut off the creation of a general-purpose endpoint. And while I was at it, I decided to shut down a whole project.
The model server is a single vLLM exposed through an OpenAI-compatible API, with each service bringing its own prompt when it calls. Beyond disclosures I wanted to summarize bills and meeting minutes too, and the agent was already building a general-purpose endpoint inside the disclosure-summary service. That structure existed to hold a boundary.
What I set out to do
Beyond disclosures, I wanted to summarize bills and meeting minutes too. Same model, same job.
But the agent’s proposal was this: add a general-purpose /api/generate endpoint to the disclosure-summary service. And it was already implementing it. I cut it off.
“I think this should be implemented on the model server side. The disclosure service shouldn’t have to care about any of this.”
Where it broke
I understand why it wanted to build it. “Multiple domains use the LLM → we need a common entry point” is a natural inference, and since the disclosure service already has code that calls the model, bolting one more thing on there looks fastest.
The problem is it already exists. vLLM serves the OpenAI-compatible /v1/chat/completions as-is. The bill-summary service brings its own system prompt, calls that directly, done.
Put a wrapper in between and this happens.
- The response format gets locked to the wrapper’s schema → awkward for other domains to use
- When vLLM ships a new feature, you have to patch the wrapper before you can use it
- Maintenance surface grows. And it grows inside a domain service
This is exactly why I split the server out in the first place. The structure existed to hold a boundary, and within two weeks a bypass almost sprouted on the inside of it.
How I fixed it
I nailed it down in three lines of rules. Before building any new LLM feature:
- Can it be done directly via the standard
/v1/chat/completions? → If yes, don’t build it - Is the domain logic genuinely complex (structured parsing, validation, multi-call orchestration)? → If not, don’t build it
- Is it common logic shared across multiple domains? → Only then consider a shared wrapper
And today that line of thinking went one step further. I asked whether the disclosure-summary service itself was even needed, whether the service API could just hold the prompt and handle it directly. Tracing back what this thing was originally supposed to do, it was “domain logic + FAISS search + prompt management,” and each piece had gone missing one by one. The prompt is better held by the caller, the API had already moved to Go, and nothing uses FAISS.
“There’s no place that uses FAISS/RAG. And looking further ahead, will the disclosure app even need FAISS search?”
The embeddings here were already loaded on the server as multilingual-e5-large, but disclosures are data you look up by key. “This company’s disclosures this quarter” is found by ticker code and period, not by semantic similarity. Wedging vector search in only makes an exact lookup inexact. RAG is for “find something similar,” not “find the exact one.”
So I decided to rip it out.
Where things stand
It’s been 18 days since I stood up the model server.
Glad I built it: separating the model server from the app; the monitoring I attached on day one (it saved me twice: there were no logs but there were metrics, and when the machine died, the minute-by-minute trajectory was still there); earlyoom and the training protocol.
Glad I didn’t build it: the general-purpose wrapper, a callback queue/gRPC layer, FAISS search, the disclosure-specific service itself.
Built it and it was wasted effort: five days of fine-tuning.
Lessons
- Half the improvement was “removing things.” And a good share of what got removed wasn’t something I set out to build; it was what the agent naturally proposed.
- The agent leans toward building something by default. Ask it and it’ll find good reasons to build. So lately my role is mostly asking, “so what gets better?”
- Don’t stack your own schema on top of a standard interface that already exists. The moment you do, upstream updates become your job.
- RAG is only for similarity problems. Finding by embedding what you could find by key is a loss.
Those three lines of criteria are only in my notes right now. They should be pinned in the project docs, but I haven’t done it yet.