The Path I Blocked as Worthless for Indexing Was My Top Referrer
Having just blocked crawlers on the sibling service, I did the same here. I blocked /api/ on the grounds that JSON has no indexing value, except the RSS feed I submitted to Naver lives under it, and Naver is the top non-direct referrer. The home page body comes from there too. I reverted 42 minutes later
I had just blocked a bot that was eating 75% of the requests on the disclosure-summary service. The bill tracker was obviously in the same state, so later the same night I went at its bot-blocking rules. On this site robots.txt is generated by code instead of served as a static file, so even adding one rule goes through code review. Among the proposed rules was one path to be blocked outright on the grounds that it had nothing worth indexing.
What I set out to do
Same night. I had just found and blocked a single backlink-tool bot eating 75% of the server’s requests on the disclosure-summary service. The bill tracker was obviously in the same state.
do the work to block crawlers that do not help user traffic, that was why dartbrief-web’s memory and network usage was high
I got the proposal back and answered short.
go with the recommended option
How I asked
The criterion was already set from the sibling service. Keep bots that create a path for a human to follow, block bots that only scrape. I had it apply that same rule, and this time clean up URL patterns as well.
On this site robots.txt is not a static file, it is code. frontend/src/app/robots.ts builds the rules and the file is emitted at build time. So adding a rule goes through code review.
Mid-task I circled back on something. There was a pull request I had parked on the sibling service, and this investigation had established it was unrelated to the memory leak.
from 249 on, we do not need to merge that right? it is confirmed that is not what causes the memory leak right?
And since referrals had come up, I asked one more thing.
hm when you say AI referrals went up does that mean users came in directly from there?
That question helped later. Telling apart a human visit from a bot request in the numbers was something I needed all night.
Where I got stuck
The proposed rules included blocking /api/. The reasoning sounded fine. That path is a proxy that hands back backend JSON, so there is nothing to index. It even had evidence attached: across 28 days of Search Console, URLs under /api/ had zero impressions. Nothing was being indexed, so there was apparently nothing to lose.
Review flagged it as a P1. It kills two things at once.
First, the RSS feed submitted to Naver lives under it. It is /api/feed/bills.xml. I submitted it to Search Advisor nine days earlier, and it is declared as an alternate link in the home document’s head. Naver is this site’s top non-direct referrer. Over 28 days it is 49 of 70 sessions. I was about to block the feed my top channel reads.
Second, the home page body comes from that path. The list, the weekly stats, and the recently passed bills are all client components, so the browser calls /api/. This is the easy part to get backwards: robots.txt does not block browsers. What a human sees is fine. The problem is crawlers that execute JavaScript. If they cannot fetch /api/, they see an empty home page.
A measurement nailed it down. The production home page’s initial HTML is 33KB, and it contains zero links to bill detail pages. Every route by which a crawler discovers a bill page runs through that fetch. A rule added to help indexing was a rule that severed it.
How I worked it out
I reverted. The rule went in at 21:30 and came back out at 22:12. Forty-two minutes, and before deploy.
Three things went in with the revert.
First, I wrote down exactly where the judgment was wrong. Raw JSON has never once been indexed. Zero impressions across 28 days of Search Console. So this was a rule that severed an existing channel to prevent a problem that did not exist. If JSON indexing ever does become a problem, the place to block it is a response header, not robots. Then it still gets crawled and just does not get indexed. The layer is different.
Second, I generalized it into the docs as a property of this site. On a site where the initial HTML is thin and the body is filled in by the client, blocking /api/ is not hiding JSON, it is hiding the body. So before adding a path to robots, check whether that path is a subresource the render depends on.
Third, I attached a regression test. It asserts that no disallow rule starts with /api. The reason it must not exist is written into the test name: the Naver RSS feed and the home page’s body fetch both go there. In six months nobody remembers this, and that is when someone asks why a JSON path is left open and blocks it again.
One more thing fell out of the same work. The bot filtering that Cloudflare used to do had been inert since the Railway migration six days earlier. When you remove a layer, the work that layer was doing disappears with it, and that was not on the migration checklist. I folded crawler blocking back into robots while I was there.
What I learned
- Before putting a path in robots, check whether the render needs it. On a site with thin initial HTML, blocking the data path blocks the body.
- robots.txt does not block browsers. Seeing the human-facing page work fine will reassure you right past the crawler staring at an empty page.
- Pick the layer you block at. To stop indexing only, use a response header. Block it in robots and the crawl itself stops, taking the feed with it.
- “Zero impressions” does not mean safe to block. The path nothing indexes can be the feed for your top referral channel.
- Leave a test that records why a reverted rule is gone. Without the reason written down, the next person adds it back for the same reason.
- When removing a layer, list what that layer was doing. Bot blocking had been quietly off for six days after the migration.