robots.txt Generator
Generate a robots.txt file visually. Configure crawl rules for major search engines and AI bots.
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
What is robots.txt Generator?
robots.txt tells web crawlers which pages they can and cannot access. It's a voluntary protocol — well-behaved crawlers respect it; malicious bots ignore it. This generator builds standard-compliant robots.txt files. For complex rules, see robotstxt.org for the full specification.
Frequently asked questions
Where should I upload robots.txt?▼
The root of your site — example.com/robots.txt. It must be at the root; subdirectories don't work.
Should I block AI bots?▼
Up to you. Allowing them means your content can be referenced by ChatGPT/Perplexity. Blocking prevents training data scraping but reduces visibility.
Related tools
Password Generator
Generate secure random passwords.
WordPress Username Generator
Generate secure WordPress usernames.
Domain Name Generator
Generate available-looking domain name ideas.
Business Name Generator
Generate brandable business name ideas.
URL Slug Generator
Convert text to URL-friendly slugs.
Lorem Ipsum Generator
Generate placeholder text in paragraphs, words, or bytes.