Definition

robots.txt

A file at the root of your domain that tells crawlers which paths they're allowed to fetch.

Full definition

robots.txt is a plain-text file at /robots.txt that uses User-agent and Allow/Disallow directives to control crawler access. It's also where you'd block or allow specific AI crawlers like GPTBot, ClaudeBot, and PerplexityBot.

Why it matters

robots.txt is the gate between AI engines and your content. Default robots.txt files typically don't address AI crawlers explicitly, leaving the door open — but many CDNs and security tools now block them by default, silently cutting off your AI-citation pipeline.

Example

User-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /

Related terms

GPTBot

OpenAI's web crawler that gathers training data for ChatGPT models.

AI Crawler

An automated bot operated by an AI company that fetches web pages for training or live answering.

llms.txt

A plain-text file at the root of your site that tells AI models which URLs are worth reading.

Put it into practice

Run a free OptimAIze scan to see how your site handles robots.txt and the rest of the GEO checklist.

Run free scan

Frequently asked questions

Is robots.txt the same as SEO?

No. robots.txt is one piece of the broader GEO (Generative Engine Optimization) program that sits on top of classical SEO. The two work together — classical SEO gets you crawled and indexed; robots.txt is part of what gets you cited by AI engines.

Do I need a tool to implement robots.txt?

For most teams, a free scanner like OptimAIze is enough to identify what's missing. Implementation is usually a copy-paste of generated markup or a small code change — no specialist tool required.