When the Constraint Changed From Money to Quota, I Needed a Queue
I shut down self-hosted GPU serving and switched to the subscription CLI, and the shared layer I had nailed down in April as "don't build it," this time I built.
I shut down self-hosted GPU serving so that three projects, the disclosure-summary service, the legislative-activity tracker and the trading bot, would share one subscription CLI. With money you just spend and read the bill later, but with quota, the moment one project burns it the others silently stop. In April I had cut off a shared layer of exactly this kind, so four months on I had to take that judgment back out and look at it again.
What I set out to do
Last week I measured the hardware’s utilization and it was 3%. Four months of tuning, and that was the result. The bottleneck wasn’t performance, it was the workload.
So I decided to flip direction. Instead of three projects sharing vLLM on my own GPU, I call claude -p on the subscription I’m already paying for every month. The disclosure-summary service does 400 a day, the legislative-activity tracker 17, the trading bot a dozen or so. Even all three combined beat a machine sitting idle.
One property changes here. The constraint changes from money to quota. It’s a subscription, so the marginal cost of a call is 0, but in exchange there’s a ceiling on how much I can use, and that ceiling is shared with my conversation sessions. With money you just spend and look at the bill later, but with quota, the moment it runs out, the window I have open right now gets blocked too.
That’s why I needed a queue.
How I asked
In April I reached exactly the opposite conclusion. When I cut off the attempt to build a common entry point because multiple domains use the LLM, I pinned down three lines of criteria. If it can be done directly through the standard interface, don’t build it; if the domain logic isn’t genuinely complex, don’t build it; only consider it when it’s common logic shared across multiple domains.
Four months later, the third condition became true for the first time. Quota metering and pacing aren’t any one project’s job. If the disclosure service burns through the quota running a backfill, the legislative tracker silently stops. It’s a state none of the three can know on its own.
So this time I built it, but I nailed down the scope first so that what went wrong in April wouldn’t recur.
The queue does three things only: recording, pacing, execution. It doesn’t need to know what a disclosure is or what a bill is.
That one line went at the very top of the project docs. April’s wrapper was bad not because it was “shared” but because it locked the response schema as its own, so every domain wanted to route around it. The queue doesn’t define a schema. Each producer decides what to send, and the queue carries it as-is.
Thanks to that, when the optimization ideas that came up later — clamping the body of periodic reports, skipping low-value disclosures — were proposed for the queue, I already had grounds to cut them. Those are the producer’s self-optimization, not the queue’s job.
Where it broke
Input per call was 33,090 tokens. That’s what you get calling claude -p with defaults. Breaking it down: tool definitions 20,661, the default system prompt 6,706, environment info and git status 3,652, project settings and the skill list 1,898, MCP definitions 173.
A summary job uses none of the tools. All of it pure loss. Turning off all four brought it to 248 tokens. A 99.3% reduction. These four flags are now pinned at the top of the docs as an absolute rule. Along with the condition that whenever the CLI version is bumped, this number must be re-measured first.
Next was deployment. I put it on Railway and it went into a crash loop. Without a Dockerfile attached, the builder picked one of the binaries under cmd/ arbitrarily, and of all things it grabbed the cleanup batch and booted without migrations. I pinned it by specifying the start command explicitly.
Then the second one. I attached a Dockerfile, but the builder field I checked through the API still showed a different value. I was stuck on it for a while, and it turned out that enum has no DOCKERFILE value at all. If a Dockerfile exists it just uses it, and the field display doesn’t change. Whether it’s actually being used can only be confirmed in the build log. The third was even more deflating — a service created through the API had no git auto-deploy hook attached, so pushing did nothing at all.
Auth snagged too. The dashboard host’s certificate sat stuck at the ownership-verification step for over an hour. DNS matched the required value and the platform’s own check passed, but it wouldn’t move on to issuance. Comparing settings against a working domain on another project showed no difference. In the end I confirmed that the CDN proxy terminates TLS with its own certificate and passes traffic behind it, so that path works fine, wrote the certificate down in the docs as unresolved, and moved on. It was a path we weren’t using.
How I fixed it
I split auth by surface. The stats dashboard is viewed by people, so email login; job submission and result lookup are called by machines, so a service token. I found out while trying to merge them into one: put email login on the API host and every machine call without a cookie gets blocked.
Splitting created one problem. A request that comes in with a service token has no email. To record “who viewed it,” the person’s identity has to be passed through, but just attaching it as a header means the outside can attach it too. So I did this.
The origin strips that header before verification and refills it only when the token is valid. An identity claim attached from outside structurally cannot get through.
The last was the graph. In the daily throughput bars, days with 0 processed still had a 2px bar. I set the height to 0, but a minimum-height setting was overriding it. It looks trivial, but “none” showing up as “a tiny bit” is the graph lying. The whole point of building the queue was to count consumption accurately, and it’s a problem if the very first screen is already off.
As an aside, I also looked into bundling multiple items into one call, and measured it too. For 5 short disclosures the input dropped 72%, but at the actual average size it was around 12%, and more than that, the tokens for a bundled response come out as one lump, so per-item values become a prorated approximation. The requirement to record usage accurately itself gets blurred. Decided not to use it.
Lessons
- “Don’t build it” is not a permanent conclusion; it’s conditional. Because I wrote it down as three lines in April, four months later I could recognize the condition had become true. Without the criteria written down, it would’ve just been me changing my story.
- A shared layer isn’t bad; a shared layer that locks the schema is bad. Make it domain-blind and there’s no reason to want to route around it.
- Measure the price of defaults. 33,090 versus 248 per call wasn’t tuning, it was just turning off what wasn’t being used. Before measuring I didn’t even know it was there.
- Check whether a setting took effect in the logs, not the settings screen. A silently ignored setting is worse than a wrong one. I got hit twice this week alone.
- Don’t decide an optimization on gains alone. Trading 12% savings for measurement accuracy is a loss on something built to measure in the first place.