Parse user-agent blocks, allow/disallow patterns, crawl-delay and sitemap directives
It’s the first file every crawler fetches. A wrong Disallow: / can wipe your entire site from Google’s index overnight — until you notice and ship a fix.
Every User-agent block with its Allow/Disallow rules, Crawl-delay directives and Sitemap references. Syntax errors are flagged separately.
Read more →Make sure your staging or pre-prod robots.txt blocks crawlers — and that your production version does not accidentally inherit those rules.
Read the docs →