AI-Aware robots.txt Generator
Explicitly allow or block GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers.
Checked = allowed. Unchecked = blocked.
# Generated by OptimAIze — https://optimaize.app/tools/robots-txt-for-ai User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / Sitemap: https://yourdomain.com/sitemap.xml
The new robots.txt rules
A robots.txt that only addresses Googlebot is no longer enough. AI crawlers each use their own user-agent, and they respect the most specific block. Use this generator to set a deliberate policy for every major engine and avoid being silently excluded — or silently scraped.
Recommended defaults
- Allow if you want to be cited in AI answers (most sites)
- Disallow if you're a paywall, exclusive publisher, or proprietary docs site
- Always include a
Sitemap:line so AI crawlers find your fresh content
Want the full picture?
This tool generates one piece. OptimAIze scans your whole site, audits structured data, content, crawler access, and answer-readiness — then gives you everything you need to be cited by AI.
Run a full GEO + AEO scan on your siteFrequently asked questions
- Why a separate robots.txt for AI?
- AI crawlers use distinct user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.). A regular robots.txt usually targets Googlebot only and silently allows or blocks AI crawlers — you should be explicit.
- Should I allow all AI crawlers?
- If you want to be cited in AI answers, yes. Blocking AI crawlers means your content can't be quoted by ChatGPT, Claude, or Perplexity. Block them only if you have a strong commercial reason to opt out.
- Will this hurt my Google ranking?
- No. Allowing GPTBot or Google-Extended doesn't change how Googlebot indexes you. Google-Extended specifically controls Bard/Gemini training, separate from Search.
- What about training vs answering?
- Most crawlers serve both purposes today and the distinction is fading. If you allow a bot, assume it can both train on your content and quote it live.