One Empty Array Turned Hitting the Limit Into an Ordinary Failure
For two hours before dawn, jobs failed across every project. The only message left was one line with the cause slot empty. And fixing it meant a number in a disclosure body could trigger a global pause
For two hours before dawn, every project’s jobs failed on the shared LLM queue. Six of the news pipeline’s daily editing jobs were caught in it and that day’s issue never went out. The cause was hitting the usage limit, and we were failing to read the envelope that delivers that news with the error array empty. Once fixed, a number in a disclosure body could trigger a global pause.
What I set out to do
From four thirty to six thirty in the morning, two hours, jobs failed regardless of project. Six daily editing jobs in the news pipeline were caught in it, and that day’s issue never went out.
What the log had was one line.
claude error(success):Nothing after the colon. And the word in parentheses is success. It says error, it says success, and there is no cause.
Where I got stuck
The CLI has an envelope shaped like this. The error flag is true, the error array is empty, and the explanation arrives in a different field. Hitting the usage limit is the representative case.
Our code only used that array for the message. So in this envelope, the message becomes an empty string.
An empty string cannot be classified. The classifier reads the message to decide whether this is a usage limit, an auth problem, or an ordinary error, and with nothing to read it lands on ordinary error.
This is where the real problem starts. Classified as a usage limit, the worker returns the job to the queue and rests for thirty minutes. Classified as an ordinary failure, it cannot take that path. It spends three retries and hardens into a permanent failure.
Those retries finish in seconds. Processing takes one or two seconds and there is no retry interval. So a job that hit the limit, instead of resting briefly and coming back, fails three times in three seconds and dies. Then that result sits in the idempotency cache. Submit the same job again and the dead result comes right back.
One empty array turned “wait a moment” into “failed permanently.”
How I worked it out
I gave the message selection an order. Look at the error array first, then the explanation field if that is empty, then standard error if that is empty too. If all three are empty, write “no message.” At least nothing after the colon stays blank.
One thing had to be separated here. Classification looks at all three fields; only the displayed message picks one. Apply the priority order to classification as well, and a fragment of body text sitting in the explanation field hides the real cause in standard error. Choosing what to show and deciding what something is are different jobs.
A few more surfaced nearby while fixing it.
If the error array holds two empty strings, joining them produces a string that is just the separator. That passes the “is it empty” check.
Broken multi-byte characters and NULs mixed into the text make the database reject the row. Then the recording itself fails and the job hardens in the in-progress state. That cleanup was needed on the success path too, not just the failure path.
String truncation was working in bytes. I changed it to cut on rune boundaries. The excerpt function especially is only called on the path where the model emitted prose, which is the path most likely to be Korean. The truncation length of 100 leaves a remainder of one when divided by three, so it was breaking every single time.
I truncated the budget-exceeded alert too. Left alone it hits Telegram’s 4,096-character limit and the alert does not go out. That happened to be the one alert a human absolutely has to see.
And the fix opened a new hole
One of the ways the classifier recognizes a usage limit was looking for the number 429. Previously it only scanned the error array, so that number would never arrive from body text.
Now it scans the explanation field too, and model output can land there. The disclosure-summary pipeline uses this queue. One mention of “429억원” in a disclosure body and the classifier reads it as a usage limit and pauses everything for thirty minutes. Retries then keep extending that pause.
So I narrowed it to nine forms that require a preceding word rather than a bare number.
After narrowing, review caught one more. The narrowed list still contained (429), and in accounting, parentheses mean a negative number. “영업손실 (429)억원” matches it directly. I removed the bare parenthetical and required a word meaning error in front. Two real notations went into the regression tests.
The auth keywords I left broad on purpose. A false positive there resolves itself once a person looks, while a miss stays unresolved.
What I learned
- When the same meaning can arrive in several fields, read all of them. We read only the error array, and hitting the limit was precisely the envelope that arrives with that array empty.
- Classifying and displaying are different jobs. Use your display priority for the decision and a body fragment in the first field hides the real cause in the last one.
- If one line of text is what separates “rest and retry” from “failed permanently,” handle the case where that line is empty. With no retry interval, three attempts is three seconds.
- Widen what a classifier scans and the data now in scope can fool it. The moment you let it read body text, numbers in body text become signals.
- False positives and misses are not worth the same. Keep auth detection broad and accept false positives; keep anything that triggers a global pause narrow. A false positive resolves itself and a miss does not.