llms.txt Generator
Build a clean llms.txt file that tells ChatGPT, Claude, Gemini, and Perplexity exactly what to read on your site.
Quick answer
An llms.txt file is a plain-Markdown brief at your domain root that tells ChatGPT, Claude, Perplexity and Gemini which pages to read and how to describe you. This generator builds a spec-correct file in seconds: add your site name, a one-line summary, and your canonical links, then copy or download the result and publish it at https://yourdomain.com/llms.txt.
# Acme Inc > Acme builds developer tools for AI search optimization. ## Docs - [Documentation](https://acme.com/docs): Full product documentation - [API Reference](https://acme.com/api): REST and webhook reference ## Optional - [Blog](https://acme.com/blog): Articles on AI search and GEO
Why llms.txt matters
AI models are now the front door to your content. llms.txt is a Markdown file at the root of your domain that points models to the canonical pages they should read first. Without it, crawlers guess — and they often guess wrong.
What goes in it
- An H1 with your site or product name
- A short blockquote summary
- A
## Docssection linking to your most important reference pages - An optional
## Optionalsection with secondary resources
A worked example
Here is what a complete, well-formed file looks like for a small SaaS. Note that every link carries a description — that sentence is what tells a model whether the page is worth fetching.
# Acme Analytics
> Acme Analytics is a privacy-first product analytics tool for B2B SaaS teams.
## Docs
- [Documentation](https://acme.com/docs): Setup, SDKs and data model
- [API Reference](https://acme.com/api): REST endpoints and webhooks
- [Pricing](https://acme.com/pricing): Plans, limits and billing FAQ
## Optional
- [Blog](https://acme.com/blog): Product analytics and privacy engineering
- [Changelog](https://acme.com/changelog): Shipped features by dateTwo more examples, different site types
A developer-docs site — the Docs section does the heavy lifting, Optional stays tiny:
# Kite DB
> Kite DB is an open-source vector database for building retrieval-augmented applications.
## Docs
- [Quickstart](https://kitedb.io/quickstart): First index in under five minutes
- [Concepts](https://kitedb.io/concepts): Embeddings, indexes and hybrid search explained
- [API Reference](https://kitedb.io/api): Client libraries and HTTP endpoints
- [Self-hosting](https://kitedb.io/self-host): Docker, Kubernetes and tuning guides
## Optional
- [Blog](https://kitedb.io/blog): Benchmarks and engineering notesAn e-commerce brand — product and policy pages matter more than docs:
# Fieldnote Goods
> Fieldnote Goods makes small-batch leather bags with a lifetime repair guarantee.
## Docs
- [Shop all bags](https://fieldnote.co/shop): Every product with materials and pricing
- [Our materials](https://fieldnote.co/materials): Full-grain leather sourcing and care
- [Shipping and returns](https://fieldnote.co/returns): Delivery times and the 60-day return policy
- [Repairs](https://fieldnote.co/repairs): Free lifetime repairs, how to submit
## Optional
- [Journal](https://fieldnote.co/journal): Craft, care guides and workshop storiesllms.txt vs robots.txt vs sitemap.xml
The three root files do different jobs. You need all three, and confusing them is the most common mistake we see in audits:
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Audience | Crawlers — permission | Search engines — discovery | LLMs — comprehension |
| Answers | "Who may fetch what?" | "What URLs exist?" | "What should you read, and what is this site?" |
| Curation | Rules, not lists | Exhaustive — every indexable URL | Selective — your 5–20 best pages |
| Format | Plain text directives | XML | Markdown |
| Replaces the others? | No | No | No — they stack |
Who actually reads llms.txt today
| Consumer | What it does with the file |
|---|---|
| AI ingestion & RAG tools | Fetch llms.txt as the entry point when crawling a domain for a knowledge base — often before the homepage. |
| Developer agents & IDEs | Read llms.txt to decide which docs pages to load into context when integrating with your product. |
| Answer engines (indirectly) | Perplexity-class engines reward the same signals llms.txt encodes: clear self-description and a curated canonical set. |
| Benchmarks & directories | Sites are scored on AI-readiness signals including llms.txt presence and quality. |
The six rules of a valid file
- Exactly one H1 at the top with the site or product name.
- A blockquote summary within a few lines of the H1 — one plain sentence.
- A
## Docssection with your most important pages. - Every link is absolute HTTPS with a short description.
- No duplicate URLs, no redirects — the file lives at the root, not behind a 301.
- Served as
text/plainortext/markdown, never HTML.
Already have one? Run it through the llms.txt Validator — it checks all six rules against a live URL or pasted text. And pair the file with an AI-aware robots.txt so the crawlers that would read it are actually allowed in; then confirm the whole picture with the AI Visibility Checker.
Common mistakes
- Linking every page. Curation is the whole point — twenty links beats two hundred.
- Relative URLs. Always use absolute https links so crawlers can resolve them.
- Serving HTML. The file must return plain text or markdown, not a rendered page.
- Writing marketing copy in the summary. State plainly what the site is, in one sentence, the way you would want a model to repeat it.
- Publishing once and forgetting. Refresh it whenever your key pages change.
After you publish
Upload to /llms.txt on your domain and run the OptimAIze scanner to confirm it's discoverable, well-formed, and actually being picked up by the AI crawlers that matter.
Control LLM Summarization and Description
An llms.txt file provides unprecedented control over how large language models (LLMs) summarize and describe your website. By explicitly stating preferred descriptions and canonical URLs, you prevent LLMs from generating generic or inaccurate summaries based solely on their training data. This ensures consistent brand messaging across AI-powered search interfaces, generative AI tools, and smart assistants, directly influencing how your digital presence is perceived and represented to users interacting with these platforms. Deploying llms.txt is a proactive SEO strategy for the AI era.
Tactical Advantages for AI-Driven Visibility
Implementing llms.txt offers distinct tactical advantages for AI-driven visibility. It allows you to specify preferred canonical URLs, preventing LLMs from indexing or referencing outdated or duplicate content. Crucially, you can provide a concise, optimized site summary (e.g., max 160 characters for optimal display) that major LLMs like ChatGPT, Claude, and Gemini will prioritize. This direct instruction bypasses the need for complex prompt engineering, ensuring your key value proposition is accurately communicated to users through AI interfaces, boosting click-through rates from AI-generated results.
Crawler Identification and Respect for Directives
LLMs interact with websites through dedicated crawlers and user agents, similar to traditional search engines. User agents such as `GPTBot`, `ClaudeBot`, `PerplexityBot`, and `Gemini-Content-Grabber` are specifically designed to honor directives found within your llms.txt file. When these crawlers encounter your `llms.txt`, they parse its contents to understand preferred descriptions, canonical links, and designated content for summarization. Disregarding these instructions can lead to LLMs generating their own, potentially suboptimal, descriptions from scraped content, which may misrepresent your brand or services.
Future-Proofing Your Digital Presence with AI Guidelines
As AI integration into search and information retrieval intensifies, establishing clear guidelines for LLMs becomes critical for long-term digital strategy. An llms.txt file acts as a direct communication channel with this emerging class of intelligent agents. By providing structured data about your site, you not only improve current AI descriptions but also future-proof your presence against evolving LLM algorithms. This positions your site to consistently convey its intended message and purpose across an increasingly AI-driven internet, maintaining brand integrity and search relevance.
Key llms.txt Directives & Crawler Specifics
| Directive | Description | Example Value | LLM Crawlers Affected |
|---|---|---|---|
| Description | A concise, preferred summary of your site for LLM display. | Max 160 characters | GPTBot, ClaudeBot, PerplexityBot, Gemini-Content-Grabber |
| Canonical | Designates the preferred URL for a given page or domain. | https://example.com/canonical-page | All major LLM crawlers |
| Sitemap | Points to your XML sitemap for comprehensive content discovery. | https://example.com/sitemap.xml | PerplexityBot, Gemini-Content-Grabber |
| User-agent | Defines specific rules for individual LLM crawlers. | User-agent: GPTBot | GPTBot |
| Allow/Disallow | Controls which parts of your site LLMs can access or ignore. | Disallow: /private/ | GPTBot, ClaudeBot |
Before Publishing Your llms.txt File:
- Ensure the file is named `llms.txt` precisely.
- Verify it's placed in your domain's root directory (e.g., `yourdomain.com/llms.txt`).
- Confirm your primary `Description` is under 160 characters for optimal display.
- Validate all `Canonical` URLs are correct and accessible.
- Test the file's accessibility by visiting `yourdomain.com/llms.txt` in a browser.
- Review any `Disallow` directives to avoid blocking crucial content.
- Check for syntax errors; even minor ones can invalidate the file.
- Consider an initial update timeline of 30-90 days post-launch.
How to Deploy Your llms.txt File:
- 1Generate Your llms.txt Content
Use our LLMs.txt Generator by inputting your site's name, a succinct one-line summary, and your critical canonical URLs. The tool will instantly compile a spec-compliant `llms.txt` file ready for deployment. Double-check all inputs for accuracy and conciseness, particularly the site description.
- 2Download or Copy the File
Once generated, you'll have the option to either directly download the `llms.txt` file or copy its contents to your clipboard. If downloading, ensure it saves with the exact filename `llms.txt` (not `llms.txt.txt` or similar). Copying is useful for pasting into an existing text editor.
- 3Access Your Website's Root Directory
Log in to your website's hosting control panel (e.g., cPanel, Plesk, FTP client, or direct SSH access). Navigate to the highest level directory of your public website, often named `public_html`, `www`, or `htdocs`. This is where your main `index.html` or `index.php` file resides.
- 4Upload or Create the File
If you downloaded the file, simply upload `llms.txt` to this root directory. If you copied the content, create a new text file named `llms.txt` in the root directory and paste the copied content into it. Save the file. Ensure no extraneous characters or empty lines are added.
- 5Verify Proper Deployment
Open your web browser and type `https://yourdomain.com/llms.txt` (replacing `yourdomain.com` with your actual domain). You should see the exact content of your `llms.txt` file displayed in the browser. If you receive a 404 error or incorrect content, recheck the file name and location.
- 6Monitor and Update Periodically
Regularly monitor how LLMs describe your site using AI search interfaces. Update your `llms.txt` whenever your brand messaging changes, new core services are offered, or significant site structure updates occur. A review cycle of every 3-6 months is recommended to ensure continued accuracy and optimal AI representation.
More questions answered
- What is an llms.txt file and why do I need one?
- An `llms.txt` file is a plain text file placed at your domain's root that provides explicit instructions to large language model (LLM) crawlers like GPTBot and ClaudeBot. You need one to directly control how these AI models summarize, describe, and link to your website, ensuring consistent branding and accurate representation in AI-powered search and generative outputs.
- Which LLMs respect the llms.txt standard?
- Major LLMs and their associated crawlers are designed to respect the `llms.txt` standard. This includes OpenAI's `GPTBot` (for ChatGPT and related services), Anthropic's `ClaudeBot` (for Claude AI), Perplexity AI's `PerplexityBot`, and Google's `Gemini-Content-Grabber` (for Gemini and AI Overviews). More LLMs are expected to adopt this standard over time.
- Where exactly should I place my llms.txt file?
- Your `llms.txt` file must be placed in the root directory of your domain. For example, if your website is `yourdomain.com`, the file should be accessible at `https://yourdomain.com/llms.txt`. Placing it in a subdirectory will prevent LLM crawlers from discovering and processing its directives.
- Can llms.txt replace my robots.txt file?
- No, `llms.txt` does not replace `robots.txt`. While both provide directives to crawlers, `robots.txt` primarily controls access for traditional search engine bots (like Googlebot) regarding crawling and indexing. `llms.txt` is specifically for guiding LLM agents on summarization, descriptions, and canonical links. They serve distinct, complementary purposes.
- What happens if I don't use an llms.txt file?
- Without an `llms.txt` file, LLMs will rely solely on their own algorithms and training data to summarize and describe your website. This can lead to generic, inaccurate, or suboptimal descriptions that don't reflect your intended brand message. You lose direct control over how your site is presented in AI-driven interfaces.
- How often should I update my llms.txt file?
- It's recommended to review and update your `llms.txt` file periodically, especially after significant changes to your website, brand messaging, or core services. A good cadence is every 3 to 6 months, or immediately if you launch a new product or pivot your main offering, to ensure LLMs always have the most current information.
- Are there any specific character limits for the description?
- Yes, while there isn't a universally enforced hard limit across all LLMs, a description between 120-160 characters is generally optimal for display in most AI interfaces. Exceeding this can lead to truncation or LLMs generating their own shorter summaries. Concise and impactful language is key.
- What's the difference between `Description` and `Canonical` directives?
- The `Description` directive provides a preferred, short summary of your website for LLM display, influencing how your site appears in AI answers. The `Canonical` directive specifies the authoritative URL for a page or domain, helping LLMs understand content hierarchy and prevent duplication issues, similar to its SEO role for traditional search engines.
Want the full picture?
This tool generates one piece. OptimAIze scans your whole site, audits structured data, content, crawler access, and answer-readiness — then gives you everything you need to be cited by AI.
Run a full GEO + AEO scan on your siteFrequently asked questions
- What is llms.txt?
- llms.txt is an emerging standard that tells AI models which pages on your site are most worth reading. It lives at the root of your domain, just like robots.txt, and is written in Markdown so LLMs can parse it easily.
- Where do I put the file?
- Upload it to the root of your domain so it's reachable at https://yourdomain.com/llms.txt. AI crawlers and ingestion tools will fetch it automatically.
- Do ChatGPT and Perplexity actually use llms.txt?
- Several AI tools and crawlers already check for it, and adoption is growing fast. Even where it isn't strictly required, publishing one helps you control how your content is summarized.
- How is llms.txt different from sitemap.xml?
- sitemap.xml lists every URL for search-engine indexing. llms.txt is curated for AI: it highlights the canonical pages you want models to learn from and links to long-form context.
- What makes a good llms.txt file?
- One H1 with your site name, a one-sentence blockquote summary, a Docs section of 5–20 absolute HTTPS links each with a short description, and an Optional section for secondary pages. Served as text/plain at your domain root — never behind a redirect, never wrapped in HTML.
- How long should the descriptions in llms.txt be?
- One sentence, 5–15 words. The description is what a model reads to decide whether to fetch the page, so say what the page covers — 'Setup, SDKs and data model', not 'Check out our docs'.
Related free tools
Each one covers a different signal AI engines read before they cite a site.