robots.txt檢查器

剖析robots.txt檔案。按user-agent分組顯示規則並標記問題。

如何使用本工具

1
Enter a site URL, for example https://example.com. If you leave off the path, the tool appends /robots.txt automatically.
2
Click Analyze. The server fetches that site's live robots.txt and returns it.
3
Read the raw file at the top, then scroll the parsed cards below to see Allow, Disallow, and Sitemap entries grouped by User-agent.
4
Adjust the URL and analyze again to compare another host or a different environment.

robots.txt是一種自願性協定，用於告訴網路爬蟲它們可以存取什麼。主流搜尋引擎會遵守它；惡意機器人則會無視它。常見錯誤包括封鎖關鍵資源、錯誤使用萬用字元以及忘記包含Sitemap指令。本檢查器會剖析任意robots.txt並按user-agent對規則進行分組。

Confirm a production site is not accidentally serving Disallow: / that blocks every crawler before a launch.
Audit a competitor's robots.txt to see which sections they keep out of search engines.
Check that your Sitemap directive is present and points at the correct sitemap URL.
Compare the robots.txt on a staging host against production to catch a stray block before deploy.
Verify that a specific bot, such as GPTBot or Bingbot, has its own group with the rules you expect.
Quickly inspect any third-party domain's crawl rules when debugging why a page is missing from search results.

robots.txt放在哪裡？▼

始終放在網域的根目錄：ejemplo.com/robots.txt。放在子目錄裡不起作用。

Disallow和noindex有什麼區別？▼

Disallow阻止爬取；noindex（在meta標籤中）阻止索引。兩者不可互換。