robots.txt检查器

解析robots.txt文件。按user-agent分组显示规则并标记问题。

如何使用本工具

1
Enter a site URL, for example https://example.com. If you leave off the path, the tool appends /robots.txt automatically.
2
Click Analyze. The server fetches that site's live robots.txt and returns it.
3
Read the raw file at the top, then scroll the parsed cards below to see Allow, Disallow, and Sitemap entries grouped by User-agent.
4
Adjust the URL and analyze again to compare another host or a different environment.

robots.txt是一种自愿性协议，用于告诉网络爬虫它们可以访问什么。主流搜索引擎会遵守它；恶意机器人则会无视它。常见错误包括屏蔽关键资源、错误使用通配符以及忘记包含Sitemap指令。本检查器会解析任意robots.txt并按user-agent对规则进行分组。

Confirm a production site is not accidentally serving Disallow: / that blocks every crawler before a launch.
Audit a competitor's robots.txt to see which sections they keep out of search engines.
Check that your Sitemap directive is present and points at the correct sitemap URL.
Compare the robots.txt on a staging host against production to catch a stray block before deploy.
Verify that a specific bot, such as GPTBot or Bingbot, has its own group with the rules you expect.
Quickly inspect any third-party domain's crawl rules when debugging why a page is missing from search results.

robots.txt放在哪里？▼

始终放在域名的根目录：ejemplo.com/robots.txt。放在子目录里不起作用。

Disallow和noindex有什么区别？▼

Disallow阻止抓取；noindex（在meta标签中）阻止索引。两者不可互换。