What I Checked When Search Console Said It Didn't Know My Disclosure Pages
Two days that started with 18 "blocked by robots.txt" reports in Naver Search Advisor and went on to Google Search Console sample inspections and measuring how thin each page was
My disclosure-summary service gets almost no search traffic — about 7 sessions a week, and zero clicks in Search Console for 30 days straight. This time I checked the reasons one by one. It isn’t a story with results yet; it’s a story about unblocking what was stuck.
What I was trying to do
Naver Search Advisor raised a warning: 18 list URLs with date and category parameters were “blocked from crawling by robots.txt”. I asked Claude whether that setting was right. It felt like things would have gone better without the block.
How I asked for it
Claude’s first answer was “wrong by today’s standards”, and it backed that up with dates.
- On September 4 I blocked the category-chip URLs (
?cat=). The web app was on Cloudflare Workers then, and the 10 ms per-request CPU limit was killing 19.4% of requests. - The next day I moved the web app to Railway and the limit went away. Only the block rule stayed, for 16 days.
The second reason for the block was “every disclosure reachable through a chip is in the sitemap anyway”. Measurement broke that too. Of the 492 files in the sitemap index, Google had fetched only 3; the 490 per-day files holding 260,000 disclosures had never been downloaded. Being in a sitemap is not the same as being discovered.
I counted what was being lost, too. Disclosure detail pages don’t link to other disclosures, so with the chips blocked, all a crawler could reach per day was the 20 disclosures on the first page of the home. With the chips open, it’s 136. I lifted the block. The chip pages have the home as their canonical, so opening them doesn’t duplicate the index.
Where it got stuck
Next was Search Console. I submitted one per-day sitemap and its status showed “Couldn’t fetch”. I read that as “file not found”. Claude queried the API directly: isPending: true, and no last-download record at all. It hadn’t failed — it just hadn’t been read yet. The on-screen label alone doesn’t tell the two apart.
Then I asked it to “pull the Search Console issues through the API and fix them”. But the Search Console API has no endpoint for index coverage or issue lists — it has only 11 methods. So Claude ran URL Inspection on a sample of 24 URLs and rebuilt the issue distribution.
- Disclosure detail pages: 9 out of 9 “URL is unknown to Google”
- Stock pages: “Discovered – currently not indexed”
Google downloads the sitemap with 2,684 stock pages every day, yet for 18 days it hadn’t fetched a single page in it. It knew about them and chose not to fetch.
There was a number that looked like the reason. Each page had only about 270 characters of unique text. The median summary was 43 characters, and 90.8% were under 100. That’s 260,000 pages on the same template differing by 270 characters each.
Progress felt painfully slow. Claude brought up the GA numbers: over four weeks, weekly sessions went from 59 to 187, and AI-assistant referrals went from 25 to 100. Only search stayed flat at 3 → 5 → 6 → 7. The service wasn’t stalled; search was.
How it got solved
I decided to make the pages thicker. Claude drew a mockup in HTML, but its table read like a log. I asked for the current disclosure’s explanation up front, with the history of earlier disclosures attached below. I also asked whether it needed more LLM calls. The history, price and correction blocks were plain DB queries, and a single parser covered the amount formats.
I also expanded the summary prompt: key points from 3 to 5–6, “why it matters” from one or two sentences to three or four. The one-line summary used in lists stayed as it was. This went out together with the job that regenerated everything for the new summary model.
The next day the page looked the same. When I asked, Claude said it had changed only the prompt, not the page. But measuring the live page showed every block in the mockup was already there. The mockup had redrawn what existed.
The real difference was one card: the disclosure-history card. Pages with it ran past 1,500 characters; pages without it were around 800. The card was missing on 76% of disclosures, because the page reused a notification-grouping field to decide “is this disclosure worth showing a history for?” That also blocked periodic reports, which aren’t notification targets — even though lining up a company’s quarterly, half-year and annual reports is itself an earnings trend.
I switched the check to disclosure type. Of 22,710 periodic reports, 88.2% now get a history card. Only the “other” type, where a history means nothing, stays blocked.
Whether indexing actually goes up, I don’t know yet. Search Console takes days to move.
Lessons
- A block rule’s reason has an expiry date. The reason goes away; the rule stays. Write “why this is blocked” next to the robots.txt line so you can revisit it later.
- Being in a sitemap is not being discovered. Check each file’s last download time in Search Console. An unread sitemap is as good as none.
- “Couldn’t fetch” may not be a failure. Tell them apart with
isPendingandlastDownloadedin the API. - The Search Console API won’t give you the issue list. Rebuilding the distribution from a URL Inspection sample is the practical route.
- Measure the current page before drawing a mockup. Just comparing character counts of rendered pages shows what’s missing, down to a single block.