#robots
3 posts tagged robots. · All tags
- What I Checked When Search Console Said It Didn't Know My Disclosure Pages Two days that started with 18 "blocked by robots.txt" reports in Naver Search Advisor and went on to Google Search Console sample inspections and measuring how thin each page was
- It Was Not AI That Came, It Was Only ChatGPT Memory and traffic jumped and analytics showed nothing. Counting the server log by User-Agent, 11,321 of 15,001 requests over three hours were one backlink-tool bot. After blocking seven bots I looked at analytics again, and the "AI Assistant, 53 sessions" bucket turned out to be one company
- The Path I Blocked as Worthless for Indexing Was My Top Referrer Having just blocked crawlers on the sibling service, I did the same here. I blocked /api/ on the grounds that JSON has no indexing value, except the RSS feed I submitted to Naver lives under it, and Naver is the top non-direct referrer. The home page body comes from there too. I reverted 42 minutes later