robots.txt Inspector
Parse robots.txt files. See rules grouped by user-agent and flagged issues.
What is robots.txt Inspector?
robots.txt is a voluntary protocol for telling web crawlers what to access. Major search engines respect it; malicious bots ignore it. Common mistakes include blocking critical resources, using wildcards incorrectly, and forgetting to include a Sitemap directive. This inspector parses any robots.txt and groups rules by user-agent.
Frequently asked questions
Where does robots.txt live?▼
Always the root of the domain — example.com/robots.txt. Subdirectories don't work.
What's the difference between Disallow and noindex?▼
Disallow prevents crawling; noindex (in a meta tag) prevents indexing. They're not interchangeable.
Related tools
Meta Tags Analyzer
Extract and analyze every meta tag from any URL.
Heading Hierarchy Analyzer
Audit H1-H6 structure for SEO and accessibility issues.
Schema Markup Validator
Extract and validate JSON-LD schema from any page.
Open Graph Inspector
Preview how a URL appears on Facebook, Twitter, and LinkedIn.
Sitemap Inspector
Analyze any XML sitemap — URL count, lastmod, structure.
Page Word Count
Count visible words on any URL.