robots.txt Inspector

robots.txt is a plain-text file served at the root of a domain (for example example.com/robots.txt) that tells web crawlers which paths they may or may not request. Defined informally since 1994 and standardized as RFC 9309 in 2022, it groups directives under one or more User-agent lines, each followed by Allow and Disallow rules plus optional Sitemap references. This inspector fetches the live file from a URL you supply and parses it line by line: it splits each line on the first colon, ignores blank lines and comments that start with #, and recognizes four directives — User-agent, Allow, Disallow, and Sitemap. It then groups those into per-crawler cards and shows the raw file (truncated at 5000 characters) above them. The parser is deliberately simple. It does not interpret wildcards, evaluate path precedence, or test whether a specific URL would be allowed for a given bot. Non-standard lines such as Crawl-delay and Host are silently skipped, and each User-agent line starts a fresh group rather than merging consecutive agents. Treat it as a fast way to read and audit what a site actually publishes, not as a compliance validator.