Shrinking the Prompt Made It 55% Faster, and Lost What Actually Mattered
The compression that produced −55% was throwing away the core of the domain. A story about a metric I wasn't measuring collapsing.
For summary quality I had gone with prompting rather than training, and the version with the system prompt cut from 277 characters to 108 had been summarizing disclosures in production for five days. Every metric I had measured looked good: processing time, JSON parsing, Opus label agreement. Then I skimmed the output by eye.
What I set out to do
Five days ago I cut the summarization prompt for gemma-4-26B-A4B from 277 characters to 108 and shipped it to production. Processing time −55%, JSON parsing 100%, Opus label agreement 73%. Every metric looked good.
Today I was just going to skim the output by eye.
Where it broke
While skimming, I noticed something odd.
“We swapped the disclosure analysis model, right? But it over-summarizes so much that nearly everything gets analyzed the same way. It doesn’t even mention things like share counts or executive names, it just summarizes at a coarser level. So the disclosure title and the content barely differ.”
Things you could tell from the disclosure title alone were being written into the summary verbatim. Attaching a summary had lost its point.
How I fixed it
The cause was the prompt I’d put in. The compressed version contains this constraint.
"summary": "<1 sentence, 50 chars max>"Tell a model to cram a whole disclosure into 50 characters and what does it throw away first? Proper nouns and numbers. Drop “who” and “how much” and the sentence shrinks fast, leaving only the skeleton of “did what.” And that skeleton is nearly identical to the disclosure title.
Crank up compression and the first thing to vanish is proper nouns, but in disclosure analysis the most important things are proper nouns and numbers. The constraint I cut for speed was aimed squarely at the core of the domain.
Worse, this wasn’t uniformly bad across all disclosures. Short, simple disclosures fit fine in 50 characters. The problem is disclosures like executive changes or share acquisitions, where names and numbers are the substance. Those all collapse into summaries like “an executive was changed.” The kind that needs summarization most was the kind most broken.
There was one more related metric. In the same validation, negative-sentiment recall was 53%. It misses half. That number was in the table back then too, and I just moved past it. The more negative a disclosure, the more “who and how much” matters, and I was using a prompt that discarded exactly that. Looking back, it connects.
While digging, the agent also went and checked the wrong thing. When I said the summary was capped at 50 characters, it pushed back that I might have misread, wasn’t I confusing it with another service’s prompt? In the end, 50 was correct. The suspicion was right, but the point it flagged was wrong. Still, that suspicion made me reopen both prompts, and that’s where the real cause turned up.
Summing up
| 108-char prompt | |
|---|---|
| Processing time | 12.84s → 5.73s ✅ |
| JSON parsing | 100% ✅ |
| Opus label agreement | 73.0% ✅ |
| Proper noun / number preservation | Worse ❌ |
| Negative-sentiment recall | 53% ❌ |
All three metrics I measured improved. And the two I didn’t look at properly got worse.
The scary part is that a metric you don’t measure doesn’t show up in the table when it gets worse. I saw −55%, called it a success, and believed that for five days. What caught it wasn’t a metric; it was skimming by eye.
The 73% Opus agreement reads differently now too. The 27% that disagreed most likely weren’t scattered randomly but concentrated in the disclosures where numbers and names matter. It’s a kind of failure the total score can’t show you.
Put another way, I was judging success by the average and failing on the distribution. I measured 200 samples without ever looking at which 200, or what types the 54 misses were. The person who built the table didn’t read the table.
Lessons
- When one metric improves, check the other side. Especially when the change is the “shrinking” kind.
- Don’t just look at the average; look at where it fails. A total of 73% tells you nothing about where the 27% is clustered.
- Decide what matters most in the domain first, and make that a metric. I wasn’t measuring “are the proper nouns still there,” so I couldn’t see it.
- Skim the output by eye now and then. This one was caught by eyeball, not by automated metrics.
- Even the AI’s wrong answers are useful. Trying to verify a wrong claim surfaced the real cause.