RevealTheme logo

robots.txt Generator

Build a basic robots.txt file from a few toggles: an explicit allow line, disallowed paths, optional blocking of five named AI crawlers, and a sitemap reference. The output is shown as text for you to copy into a file named robots.txt.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

How to use this tool

  1. 1

    Tick 'Allow all standard crawlers' to add an explicit Allow: / line, or untick it to leave it out.

  2. 2

    Enter the paths you want crawlers to skip in the disallowed box, one per line (for example /admin/ or /api/).

  3. 3

    Optionally tick 'Block AI training crawlers' to add Disallow rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, and paste your sitemap URL.

  4. 4

    Select the generated text in the preview box, copy it, and save it as a file named robots.txt at your domain root.

What does a robots.txt file actually control?

robots.txt is a plain-text file served from your domain root that tells web crawlers which URL paths they may request. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022, using User-agent lines to name a crawler and Disallow / Allow lines to grant or deny path prefixes. Compliance is voluntary: reputable crawlers like Googlebot read and obey it, but nothing forces a misbehaving bot to honor it. The crucial gotcha is that robots.txt controls crawling, not indexing. If you Disallow a page but other sites link to it, Google may still index the URL and list it in results with 'no information is available for this page.' To truly keep a page out of search, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead. This tool builds the file by simple string assembly: a single User-agent: * group, your typed Disallow lines verbatim, optional per-bot blocks, and a Sitemap line. It does no validation, so whatever you type is pasted as-is.

Common use cases

  • Keeping crawlers out of admin, API, or internal paths like /admin/ and /api/ while leaving the rest of the site open.

  • Generating a starter robots.txt for a brand-new site that does not have one yet.

  • Blocking the five named AI training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) so your content is not scraped for model training.

  • Adding or correcting the Sitemap: line so search engines can discover your sitemap.xml.

  • Drafting rules quickly to paste into a CMS or static-site config, then hand-editing for anything advanced.

  • Teaching teammates what a minimal, readable robots.txt looks like before they edit the real one.

Frequently asked questions

Is my input uploaded anywhere?▼
No. The file is assembled entirely in your browser using local React state. Nothing you type in the disallowed paths or sitemap fields is sent to a server.
Where should I put the robots.txt file?▼
At your domain root, reachable as example.com/robots.txt. Crawlers only look there; a file in a subdirectory such as example.com/blog/robots.txt is ignored.
Does this validate my rules or guarantee they are correct?▼
No. The generator does plain string concatenation and pastes your disallowed paths verbatim, with no syntax checking. Review the output yourself, and test it in Google Search Console's robots.txt report before relying on it.
Will a Disallow rule hide a page from Google search?▼
Not necessarily. Disallow blocks crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header.
Which AI crawlers does the block option cover?▼
Exactly five: GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended. Each gets its own User-agent group with Disallow: /. Other AI bots such as Bytespider or Amazonbot are not included, so add those by hand if you need them.
Can I write wildcard, crawl-delay, or per-search-engine rules here?▼
Not through the toggles. This tool only emits a single User-agent: * group plus the optional AI-bot blocks. For patterns like * and $, Crawl-delay, or rules targeting specific search engines, edit the generated file manually or consult the full spec at robotstxt.org.
Should I block AI crawlers?▼
It is a tradeoff. Allowing them lets your content be referenced by assistants like ChatGPT and Perplexity; blocking aims to keep it out of training data but relies on those bots honoring robots.txt, which is voluntary.

Related tools

Password Generator

Generate secure random passwords.

WordPress Username Generator

Generate secure WordPress usernames.

Domain Name Generator

Generate available-looking domain name ideas.

Business Name Generator

Generate brandable business name ideas.

URL Slug Generator

Convert text to URL-friendly slugs.

Lorem Ipsum Generator

Generate placeholder text in paragraphs, words, or bytes.

Subscribe for updates

We do not sell your email. We do not spam.

© 2026 RevealTheme. All rights reserved.