#incident-response
6 posts tagged incident-response. · All tags
- A PR That Verified Zero Missing Rows Stopped Analysis for Three Hours With One Comment Line The PR for regenerating 260,000 summaries accounted for locks, rolling deploys and every stray row. One format placeholder inside a comment in the last commit broke the SQL, and all four tests still passed
- One Empty Array Turned Hitting the Limit Into an Ordinary Failure For two hours before dawn, jobs failed across every project. The only message left was one line with the cause slot empty. And fixing it meant a number in a disclosure body could trigger a global pause
- I Lost 2.5 Months of Logs in My First Hour Back In the first DGX Spark session in two months, "check the logs" turned into a container recreate and 334MB vanished. What survived was 14 days that another system happened to be watching
- I Wiped Every Docker Volume: The AI Warned Me, and I Deleted It Anyway After deleting Docker.raw and losing every local postgres, I built shared infrastructure instead of trying to recover
- I Set a Container Memory Limit and the Whole Machine Froze Anyway A 15-hour incident where Docker memory limits did nothing on a DGX Spark and a 26B LoRA run froze the whole machine, plus the two-layer defense I built
- Two AIs Blamed Each Other: What Happens When Every Project Gets Its Own Agent Each project's agent looked only at its own logs and pleaded innocent, so I laid both logs side by side to reach a conclusion