تولیدکننده robots.txt

نحوه استفاده از این ابزار

1. 1

   Tick 'Allow all standard crawlers' to add an explicit Allow: / line, or untick it to leave it out.

2. 2

   Enter the paths you want crawlers to skip in the disallowed box, one per line (for example /admin/ or /api/).

3. 3

   Optionally tick 'Block AI training crawlers' to add Disallow rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, and paste your sitemap URL.

4. 4

   Select the generated text in the preview box, copy it, and save it as a file named robots.txt at your domain root.

تولیدکننده robots.txt چیست؟

موارد استفاده رایج

• Keeping crawlers out of admin, API, or internal paths like /admin/ and /api/ while leaving the rest of the site open.

• Generating a starter robots.txt for a brand-new site that does not have one yet.

• Blocking the five named AI training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) so your content is not scraped for model training.

• Adding or correcting the Sitemap: line so search engines can discover your sitemap.xml.

• Drafting rules quickly to paste into a CMS or static-site config, then hand-editing for anything advanced.

• Teaching teammates what a minimal, readable robots.txt looks like before they edit the real one.

پرسش‌های متداول