robots.txt產生器

以視覺化方式產生robots.txt檔案。為主流搜尋引擎和AI機器人設定爬取規則。

允許所有標準爬蟲封鎖 AI 訓練爬蟲

禁止路徑 (每行一個)

網站地圖 URL

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

如何使用本工具

1
Tick 'Allow all standard crawlers' to add an explicit Allow: / line, or untick it to leave it out.
2
Enter the paths you want crawlers to skip in the disallowed box, one per line (for example /admin/ or /api/).
3
Optionally tick 'Block AI training crawlers' to add Disallow rules for GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, and paste your sitemap URL.
4
Select the generated text in the preview box, copy it, and save it as a file named robots.txt at your domain root.

robots.txt告訴網路爬蟲哪些頁面可以存取、哪些不可以。它是一種自願性協定：行為規範的爬蟲會遵守它；惡意機器人則會無視它。本產生器建立符合標準的robots.txt檔案。對於複雜規則，請查閱robotstxt.org和完整的規範。

Keeping crawlers out of admin, API, or internal paths like /admin/ and /api/ while leaving the rest of the site open.
Generating a starter robots.txt for a brand-new site that does not have one yet.
Blocking the five named AI training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) so your content is not scraped for model training.
Adding or correcting the Sitemap: line so search engines can discover your sitemap.xml.
Drafting rules quickly to paste into a CMS or static-site config, then hand-editing for anything advanced.
Teaching teammates what a minimal, readable robots.txt looks like before they edit the real one.

我應該把robots.txt上傳到哪裡？▼

上傳到你網站的根目錄：example.com/robots.txt。它必須位於根目錄；放在子目錄裡不起作用。

我應該封鎖AI機器人嗎？▼

這取決於你。允許它們意味著ChatGPT/Perplexity可以引用你的內容。封鎖它們可避免訓練資料被爬取，但會降低能見度。