A PR That Verified Zero Missing Rows Stopped Analysis for Three Hours With One Comment Line
The PR for regenerating 260,000 summaries accounted for locks, rolling deploys and every stray row. One format placeholder inside a comment in the last commit broke the SQL, and all four tests still passed
I shipped a change to regenerate all 260,000 summaries in my disclosure-summary service. The PR counted every row to prove nothing would be missed and even measured migration locks. Right after the deploy, not a single new disclosure got a summary. The cause was one format placeholder Claude had put inside an SQL comment. This episode is about an outage the AI caused and the AI fixed.
What I was trying to do
The day before, I had moved the summary model from haiku to sonnet. Cross-checking 1,360 supply-contract summaries against financial data showed 9.2% of them shifting the amount by a digit — the error crept in while converting won into hundred-millions of won.
Upgrading the model only changes disclosures that come in from now on. So I decided to regenerate the existing summaries too: 261,404 of them.
How I asked for it
The PR was careful. Codex review kept raising P1 issues, and each got fixed.
- Putting a constraint swap and a 70,000-row UPDATE in one file would block reads for 16.7 seconds. It was split into three files.
- A path where jobs submitted by an old instance with the old prompt during a rolling deploy would be skipped forever was closed.
- Selecting targets by a list of model names was dropped after it turned out to miss 249 rows. The rule was flipped to “every summary already completed is outdated”, and every row was classified to confirm zero misses.
Right before merging, I asked one thing: were the rows not yet redone with the new model being processed newest first? They weren’t. Reanalysis enqueued the newest disclosures first, but the query that claims work picked the oldest first. I said “fix the ordering too and push it now”, and a commit splitting the sort key per work pool went on last. Deployed at 12:56.
Where it got stuck
The SQL was assembled in Go with Sprintf. A comment explaining the sort keys contained the placeholder %[4]s, and the value substituted there spanned two lines.
-- ── the last two keys differ per pool (CASE WHEN status = 'PENDING' THEN created_at ...,
rcept_no). ──────── ← SQL from here onA -- comment ends at the end of the line. The second line escaped the comment and became SQL, and the leftover closing parenthesis broke the syntax. The claim query failed every minute. Disclosures kept coming in, but no summaries were attached, and the home page only shows disclosures that have a summary. So the site looked frozen.
All four tests passed. They only checked whether certain strings appeared in the generated SQL. The broken SQL contained every one of them.
There was a signal 35 minutes after the deploy. While I was asking about something else, Claude checked whether the new prompt had taken effect. The answer was “0 so far, 6 PENDING”, and the last completion time was right there in the table. Claude read it as “not done yet” and moved on. It looked at what was coming in, not at what was going out.
How it got solved
Two hours later, I noticed first.
Why does it look like no new disclosures have shown up since 11:36? Can you check?
The first query was wrong too — it converted the time zone backwards, so it looked like ingestion had stopped. Once corrected, ingestion was fine; analysis had simply completed nothing for 3 hours 24 minutes, with 88 PENDING piled up. Following the claim failed log led to syntax error at or near ".". That was the first time Claude said, “My PR is the cause.”
The stumbling continued during the fix. A verification command wrapped in double quotes let the shell eat $1–$7. A command to revert local changes also wiped the fix it had just made, so it was redone.
The fix itself was simple: take the placeholder out of the comment and describe it in words. Four variants of the query were run through PREPARE on a real Postgres 16 to confirm the syntax. A regression test was added that strips comment lines and checks that the parentheses balance — the broken version has one extra closing parenthesis and fails it. Merged at 15:49; a few minutes later PENDING was dropping from 88 to 58.
Lessons
- Don’t put placeholders inside comments of SQL you build from strings. Nothing guarantees the value will be one line.
- Validate generated SQL as SQL, not as a string. A test that checks “is this fragment present” passes even when the syntax is broken. Running
PREPAREonce against a real database is the cheapest check. - The last commit of a big PR is the least verified. Be separately suspicious of the fix rushed in right before merge, after the main body went through several review rounds — even when the one rushing it is you.
- Check a deploy by what goes out, not what comes in. “There are PENDING rows” can mean normal or broken. “Has anything completed since the deploy?” settles it in one query.
- A human noticing something off on screen beat the monitoring. This path had no “zero completions for N minutes” alert.