robots.txt Generator
Build a robots.txt with per-bot rules, AI crawler blocking, and sitemap links
User-agent: * Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
About this tool
The robots.txt Generator builds a complete robots.txt file from per-bot User-agent rules — Allow and Disallow paths, optional crawl-delay, and sitemap links — with a one-click preset to block the most common AI training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot, and others) by name. robots.txt remains the standard, actually-honored mechanism for controlling crawler access: OpenAI and Anthropic both officially point to robots.txt (not the newer, largely unsupported llms.txt proposal) for managing how their crawlers interact with your site.
When to use it
- →Setting up a robots.txt for a new site with sensible defaults for admin/API paths
- →Blocking specific AI training crawlers (GPTBot, Google-Extended, CCBot) while still allowing search engine indexing
- →Declaring sitemap locations so crawlers discover your full page list efficiently
- →Building different crawl rules for different bots — e.g. stricter rules for aggressive scrapers than for Googlebot
Tips
- ◆One User-agent group can list multiple bots on separate lines above shared Allow/Disallow rules — you don't need a separate group per bot unless the rules actually differ.
- ◆robots.txt is a request, not enforcement — well-behaved crawlers (Googlebot, Bingbot, and the major AI crawlers) respect it, but nothing stops a scraper from ignoring it entirely.
- ◆Use the 'Block AI crawlers' preset as a starting point, then remove any bots you actually want indexing your content for AI-powered search or citation.
Frequently asked questions
Does blocking AI crawlers in robots.txt actually work?
For crawlers that respect the standard — OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended — yes, since both OpenAI and Anthropic have officially stated they honor robots.txt for managing crawler access. It won't stop crawlers that don't respect the file at all, which does happen with some more aggressive or less reputable scrapers.
Should I use llms.txt instead of or alongside robots.txt?
As of 2026, llms.txt has real adoption among documentation platforms but is not honored by the major AI systems — Google has stated it has no plans to support it, and OpenAI/Anthropic point to robots.txt instead. robots.txt is the mechanism that actually controls crawler access today; llms.txt does not reliably improve AI visibility.
What's the difference between Disallow and noindex?
Disallow in robots.txt tells a crawler not to fetch a path at all — but if another page links to it, search engines may still index the URL without visiting it, showing it in results with no description. A noindex meta tag (set on the page itself) explicitly tells search engines not to include the page in results, but requires the crawler to fetch the page to see the tag — the two mechanisms solve different problems and are sometimes used together incorrectly.
Do I need a separate robots.txt for each subdomain?
Yes — robots.txt only applies to the exact host and scheme it's served from. A file at example.com/robots.txt has no effect on blog.example.com or a different subdomain; each needs its own robots.txt at its own root.