llms.txt Implementation Guide with Examples

The short answer

Implement llms.txt by publishing a concise Markdown file at /llms.txt that names the site, explains its purpose, and links to a curated set of canonical, useful pages. Keep robots.txt and sitemap.xml separate, avoid copying whole pages into the file, validate every URL, and update the list when important content moves.

Key takeaways

llms.txt is a curated discovery document, not an access-control file or a replacement for sitemap.xml.

A short list of canonical, high-signal URLs is more useful than an inventory of every page.

Each linked page still needs crawl access, stable HTML, a canonical URL, and a direct answer.

Validate status codes, redirects, duplicates, and content type before publishing.

Treat llms.txt as maintained documentation and review it whenever the site structure changes.

What belongs in llms.txt?

The file should orient an AI system quickly: what the site is, which pages are authoritative, and where to find core documentation or policies. Include durable pages that explain the organization, product, documentation, pricing, important concepts, and changelog. Exclude thin archives, tracking URLs, duplicates, filters, and pages you would not want summarized.

Include	Usually exclude	Reason
Canonical product and documentation pages	Search and filter results	Stable pages provide reusable context
Glossary and high-quality guides	Tag archives and pagination	Definitions and procedures are easy to extract
Current pricing, policies, and changelog	Expired campaigns	Time-sensitive facts need an authoritative URL
A small number of representative examples	Every near-duplicate location page	Curation reduces ambiguity

What is a practical llms.txt format?

Use ordinary Markdown: one H1, a short summary, optional notes, then descriptive links grouped by purpose. Keep labels meaningful outside the surrounding page. The proposed convention is intentionally readable by both people and machines, so avoid scripts, hidden directives, or invented ranking commands.

# Example Company

> Official documentation and product information for Example Company.

## Core
- [Product overview](https://example.com/product): Capabilities and supported use cases.
- [Documentation](https://example.com/docs): Setup and API reference.
- [Pricing](https://example.com/pricing): Current plans and limits.

## Learn
- [Glossary](https://example.com/glossary): Definitions of domain terms.

How do you publish and validate the file?

Inventory the pages that contain your most authoritative answers.
Remove redirected, duplicated, private, outdated, and low-value URLs.
Group the remaining links by visitor purpose and write a specific description for each.
Publish the plain-text Markdown response at the exact root path /llms.txt.
Fetch the public URL, follow every link, and confirm each target is canonical and indexable.
Add ownership and review cadence to the same workflow that updates navigation and sitemaps.

Do not claim that llms.txt grants access or guarantees citations. Crawlers still obey their own policies, robots.txt, network controls, and retrieval systems.

How should llms.txt be deployed on different stacks?

Stack	Recommended method	Verification
Static site	Commit a public/llms.txt file	Open the root URL after deployment
CMS	Use a root-level route, plugin, or edge rule	Confirm no theme HTML wraps the response
Headless app	Serve a plain-text route from the public origin	Test production, not only preview
Large documentation site	Generate from an approved canonical registry	Diff changes and reject broken links in CI

Which llms.txt mistakes reduce its usefulness?

Listing hundreds of URLs without descriptions or hierarchy.
Linking to redirects, staging hosts, tracking parameters, or non-canonical duplicates.
Copying robots directives into llms.txt and assuming they control access.
Publishing marketing claims instead of neutral descriptions of what each page contains.
Generating the file once and leaving deleted or outdated pages in it.
Serving an HTML error page, authentication screen, or wrong content type at the path.

How can you measure whether llms.txt is helping?

Measure the chain, not a promised ranking lift. First confirm the file is fetched. Then inspect whether the linked pages receive crawler visits, appear in retrieval citations, and are summarized accurately. Compare a fixed prompt panel before and after publication, while recording other changes so llms.txt is not credited for work performed by content or schema improvements.

Frequently asked questions

Is llms.txt an official web standard?

llms.txt is a community proposal rather than a universally enforced web standard. Publish it as a low-risk discovery aid, but keep established controls such as robots.txt, sitemaps, canonicals, and accessible HTML correct.

Does llms.txt replace robots.txt?

No. robots.txt communicates crawl permissions, while llms.txt curates useful pages and context. A link in llms.txt cannot override a robots rule, firewall block, login requirement, or inaccessible page.

How many URLs should llms.txt contain?

There is no universal number. Include the smallest set that represents the site accurately. A focused file with descriptive canonical links is easier to maintain and interpret than a complete URL dump.

Should llms.txt include every blog post?

Usually not. Include cornerstone articles, durable research, and key guides. Let sitemap.xml provide broad discovery, and use llms.txt to point at the pages that best explain the site.

How do I test llms.txt?

Fetch /llms.txt from production, confirm a successful plain-text response, then check every linked URL for status, final destination, canonical, language, and useful server-rendered content.

How to implement llms.txt correctly