llms.txt
A plain-text file at the root of your site that tells AI models which URLs are worth reading.
Full definition
llms.txt is a community standard (originally proposed by Jeremy Howard) for a Markdown-formatted file served at /llms.txt. It lists the most important URLs on your site — docs, key landing pages, changelogs — with short descriptions so AI models can find the high-signal content quickly without crawling everything.
AI models have limited context windows. When they decide what to ingest from your site, llms.txt is a curated short list that vastly outperforms whatever they'd discover from sitemap.xml alone. Sites with a well-maintained llms.txt get cited disproportionately often.
Example
/llms.txt → # Acme\n\n## Docs\n- [Getting started](https://acme.com/docs/start): 5-minute setup guide\n- [API reference](https://acme.com/docs/api): Full REST API spec
Related terms
Generative Engine Optimization — optimizing for citation by AI chat, search, and assistants.
Answer Engine Optimization — optimizing for direct-answer surfaces like AI assistants and voice search.
Google's AI-generated answers that appear above the classical search results.
OpenAI's web crawler that gathers training data for ChatGPT models.
Put it into practice
Run a free OptimAIze scan to see how your site handles llms.txt and the rest of the GEO checklist.
Run free scanFrequently asked questions
Is llms.txt the same as SEO?
No. llms.txt is one piece of the broader GEO (Generative Engine Optimization) program that sits on top of classical SEO. The two work together — classical SEO gets you crawled and indexed; llms.txt is part of what gets you cited by AI engines.
Do I need a tool to implement llms.txt?
For most teams, a free scanner like OptimAIze is enough to identify what's missing. Implementation is usually a copy-paste of generated markup or a small code change — no specialist tool required.
Understanding the Role of llms.txt in AI Search
The `llms.txt` file serves as a crucial signaling mechanism, guiding large language models (LLMs) and other AI agents on how to interact with your website's content. Similar in concept to `robots.txt` for traditional search engines, `llms.txt` provides directives for AI-driven crawlers and data ingestors. By explicitly stating which sections of your site are suitable for AI consumption, training, or citation, you gain a degree of control over your digital footprint in the AI landscape. It helps prevent AI models from ingesting outdated, irrelevant, or sensitive content, ensuring that only curated, high-quality information contributes to your AI visibility and reputation. Implementing it is a proactive step in managing your site's presence in generative AI responses.
Strategic Implementation for Optimal AI Visibility
Effective implementation of `llms.txt` goes beyond mere presence; it requires strategic thinking about which content truly benefits your AI search visibility. Prioritize publicly accessible content that is authoritative, factual, and relevant to your core offerings. For instance, detailed product specifications, comprehensive knowledge base articles, or thought leadership pieces are excellent candidates for `Allow` directives. Conversely, sections like user comments, login pages, or dynamically generated content that lacks permanent value should be considered for `Disallow`. A well-crafted `llms.txt` ensures AI models focus their processing power on your most valuable assets, leading to more accurate citations and a stronger presence in AI-generated answers and summaries.
Syntax and Best Practices for llms.txt Directives
The `llms.txt` file uses a simple, line-based syntax, primarily relying on `User-agent`, `Allow`, and `Disallow` directives. It's imperative to specify `User-agent: *` to apply rules broadly to all AI models, or target specific agents if known (e.g., `User-agent: Google-Extended`). Paths specified in `Allow` or `Disallow` should be relative to the root of your domain. Always place `llms.txt` in your domain's root directory (e.g., `https://www.example.com/llms.txt`). Best practices include keeping the file concise, avoiding conflicting rules, and regularly reviewing its content as your website evolves. Testing your directives, perhaps with specialized AI crawler simulators, can help confirm correct interpretation by consuming models.
llms.txt Directives & Examples
| Directive | Purpose | Example Use Case |
|---|---|---|
| User-agent: | Identifies the AI model or bot the following rules apply to. | User-agent: * |
| Allow: | Permits AI models to access and potentially ingest content at the specified path. | Allow: /knowledgebase/ |
| Disallow: | Forbids AI models from accessing or ingesting content at the specified path. | Disallow: /user-reviews/ |
| Crawl-delay: | Suggests a delay between successive requests to prevent server overload (less common for LLMs). | Crawl-delay: 10 |
| Sitemap: | Points to the XML sitemap(s) for AI models to discover content (optional but useful). | Sitemap: https://www.example.com/sitemap.xml |
Essential llms.txt Optimization Checklist
- Verify `llms.txt` is located at your website's root directory.
- Ensure all critical, publicly available content is explicitly 'Allow'ed.
- Disallow sensitive, private, or low-quality content that shouldn't inform AI models.
- Use `User-agent: *` for broad AI model coverage unless specific agents require custom rules.
- Regularly review and update your `llms.txt` file as your site's content changes.
- Confirm syntax is correct and free of errors to ensure proper interpretation by AI crawlers.
Implementing Your First llms.txt File
- 1Create the File
Open a plain text editor and save an empty file as `llms.txt`.
- 2Define Directives
Add `User-agent: *` followed by your desired `Allow:` and `Disallow:` paths, one directive per line.
- 3Upload to Root
Upload this `llms.txt` file to the root directory of your website (e.g., `public_html` or `www`).
- 4Verify Access
Check that the file is publicly accessible via your browser at `https://www.yourdomain.com/llms.txt`.
More questions answered
- What is the primary purpose of llms.txt?
- The primary purpose of `llms.txt` is to guide large language models and other AI agents on which parts of your website they should or should not access, ingest, or use for training and citation, similar to how `robots.txt` functions for traditional search engine crawlers.
- Where should I place my llms.txt file?
- Your `llms.txt` file must be placed in the root directory of your domain. For instance, if your website is `example.com`, the file should be accessible at `https://example.com/llms.txt` for AI models to discover and interpret it.
- Can llms.txt prevent all AI models from accessing my content?
- While `llms.txt` provides strong directives, it's a voluntary protocol. Well-behaved AI models will adhere to it, but it does not technically 'block' access. It relies on the good faith and programming of the AI agent creators to respect your specified rules and control AI visibility.
- What happens if I don't have an llms.txt file?
- If you don't have an `llms.txt` file, AI models generally assume that all publicly accessible content on your site is fair game for ingestion, training, or citation. This could lead to AI models using content you might prefer to keep out of AI-generated responses.
Explore further
Connected guides to keep going — short reads, all internally linked.