Инспектор robots.txt

Как пользоваться этим инструментом

1. 1

   Enter a site URL, for example https://example.com. If you leave off the path, the tool appends /robots.txt automatically.

2. 2

   Click Analyze. The server fetches that site's live robots.txt and returns it.

3. 3

   Read the raw file at the top, then scroll the parsed cards below to see Allow, Disallow, and Sitemap entries grouped by User-agent.

4. 4

   Adjust the URL and analyze again to compare another host or a different environment.

Что такое инспектор robots.txt?

Типичные сценарии использования

• Confirm a production site is not accidentally serving Disallow: / that blocks every crawler before a launch.

• Audit a competitor's robots.txt to see which sections they keep out of search engines.

• Check that your Sitemap directive is present and points at the correct sitemap URL.

• Compare the robots.txt on a staging host against production to catch a stray block before deploy.

• Verify that a specific bot, such as GPTBot or Bingbot, has its own group with the rules you expect.

• Quickly inspect any third-party domain's crawl rules when debugging why a page is missing from search results.

Часто задаваемые вопросы