robots.txt জেনারেটর

এই টুলটি কীভাবে ব্যবহার করবেন

1. 1

   Tick 'Allow all standard crawlers' to add an explicit Allow: / line, or untick it to leave it out.

2. 2

   Enter the paths you want crawlers to skip in the disallowed box, one per line (for example /admin/ or /api/).

3. 3

   Optionally tick 'Block AI training crawlers' to add Disallow rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, and paste your sitemap URL.

4. 4

   Select the generated text in the preview box, copy it, and save it as a file named robots.txt at your domain root.

robots.txt জেনারেটর কী?

সাধারণ ব্যবহারের ক্ষেত্র

• Keeping crawlers out of admin, API, or internal paths like /admin/ and /api/ while leaving the rest of the site open.

• Generating a starter robots.txt for a brand-new site that does not have one yet.

• Blocking the five named AI training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) so your content is not scraped for model training.

• Adding or correcting the Sitemap: line so search engines can discover your sitemap.xml.

• Drafting rules quickly to paste into a CMS or static-site config, then hand-editing for anything advanced.

• Teaching teammates what a minimal, readable robots.txt looks like before they edit the real one.

সচরাচর জিজ্ঞাসিত প্রশ্ন