How to add llms.txt to Astro
Astro serves anything in /public unchanged. Either drop llms.txt there or use an endpoint for dynamic content.
Before you start
- ✓An Astro project (static or SSR adapter)
- ✓An llms.txt body — static file or imported module
Install in 3 steps
- 1
Static option
Place llms.txt in /public. Astro copies it untouched at build.
- 2
Dynamic option
Create src/pages/llms.txt.ts exporting a GET that returns new Response(content, { headers: { 'Content-Type': 'text/plain' } }).
- 3
Verify
Build and preview. Confirm headers.
Troubleshooting
Set prerender = true in the endpoint to render at build and avoid runtime cost.
Confirm /public is part of your build output — some custom Astro configs override the default.
Common gotchas
- • If you're on an SSR adapter, set prerender = true for the endpoint to avoid runtime cost
Generate your llms.txt in 30 seconds
Use the free OptimAIze generator, then follow the steps above to deploy on Astro.
Frequently asked questions
Does Astro need both llms.txt and robots.txt?
Yes. robots.txt grants crawler permission; llms.txt curates which pages matter. Together they form the minimum AI-search setup on any Astro site.
Will adding llms.txt slow down my site?
No. It's a tiny static text file fetched once and cached aggressively. Page-load impact is effectively zero.
How often should I update llms.txt?
Whenever your canonical content set changes — new docs section, new product line, new pricing page. A monthly review is a sensible cadence for most sites.
Can I see whether AI engines read my llms.txt?
Check your server logs for user agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended hitting /llms.txt. Most sites see traffic within days of publishing.
Other platforms
Integrating llms.txt into Astro Project Structure
For an Astro project, the `llms.txt` file should be placed in the `public/` directory. This ensures it's served statically at the root of your domain, like `yourdomain.com/llms.txt`, making it discoverable by large language models. Astro’s build process will copy this file directly to the output directory without any special configuration. This placement is critical for allowing AI agents and crawlers to consistently locate and interpret your LLM directives, ensuring proper content attribution and usage. Confirming its accessibility post-deployment is a vital step for effective AI visibility management.
Crafting Effective LLM Directives for Astro Content
Within `llms.txt`, each directive needs to be precisely crafted to guide LLM behavior. For instance, `Allow: /blog/` permits AI models to cite or summarize content within your blog section, while `Disallow: /private-docs/` prevents such usage for sensitive areas. Utilize specific paths to grant or restrict access to particular Astro routes or components. Consider granular controls for dynamically generated content, ensuring that even pages rendered on demand are appropriately signaled. Regular audits of these directives against your content strategy are essential to maintain optimal AI visibility and content control.
Automating llms.txt Deployment and Validation in Astro
While manual placement is straightforward, larger Astro sites benefit from automated `llms.txt` deployment. Consider integrating a build script that generates or updates the file during your CI/CD pipeline, especially if directives change based on staging or production environments. Post-deployment validation is crucial; use tools or simple fetch requests to confirm `llms.txt` is served correctly and its contents are as expected. This proactive approach ensures continuous LLM compliance and prevents accidental misconfigurations that could impact your AI search visibility.
llms.txt Directives for Astro Projects
| Directive | Usage Example | Impact on AI Visibility |
|---|---|---|
| Allow: / | Permits AI models to process all public content on your Astro site. | Maximizes visibility, but requires careful content review for AI suitability. |
| Disallow: /marketing-assets/ | Prevents AI from using specific directories, e.g., proprietary graphics or campaigns. | Protects specific assets from AI ingestion, reducing intellectual property risks. |
| Crawl-Delay: 10 | Requests AI crawlers to wait 10 seconds between page requests from your Astro site. | Manages server load during AI crawling, improving site stability. |
| Noindex: /temp-pages/ | Instructs AI not to index content within temporary or development paths. | Prevents unready or irrelevant content from appearing in AI responses. |
| User-agent: SpecificBot | Targets directives to a particular AI agent (e.g., GPTBot or Bard-Crawler). | Allows fine-grained control over how individual AI models interact with your Astro content. |
Essential llms.txt Astro Integration Checklist
- Verify `llms.txt` is in the `public/` directory for root-level accessibility.
- Ensure the file path is accessible at `yourdomain.com/llms.txt` post-deployment.
- Define clear `Allow` and `Disallow` rules for all critical Astro routes.
- Include a `User-agent: *` block for general directives, then specific ones as needed.
- Review and update `llms.txt` whenever significant content changes or new Astro routes are added.
- Test accessibility and content of `llms.txt` using browser or `curl` after each deployment.
Implementing llms.txt on an Astro Site
- 1Create File
In your Astro project root, navigate to the `public/` folder. Create a new text file named `llms.txt` within this directory.
- 2Add Directives
Open `llms.txt` and add your directives, such as `User-agent: *` followed by `Allow: /` or specific `Disallow` rules for sensitive content paths in your Astro site.
- 3Deploy Site
Build and deploy your Astro project as usual. The `llms.txt` file from the `public/` directory will be included and served at the root of your domain.
- 4Validate Accessibility
After deployment, open your browser and navigate to `yourdomain.com/llms.txt` to confirm the file is publicly accessible and its contents match your intended directives for AI models.
More questions answered
- What is the primary purpose of llms.txt for an Astro website?
- The primary purpose of `llms.txt` for an Astro website is to communicate explicit instructions to large language models and AI crawlers regarding which content they are permitted or restricted from using for training, summarization, or citation. This helps manage your site's AI visibility and intellectual property.
- Where should llms.txt be placed in an Astro project?
- In an Astro project, the `llms.txt` file must be placed in the `public/` directory. This ensures that when the site is built and deployed, the file is served at the root level of your domain, making it discoverable by AI agents at `yourdomain.com/llms.txt`.
- Can I use wildcards in llms.txt directives for Astro paths?
- Yes, standard wildcard characters are supported in `llms.txt` directives, similar to `robots.txt`. For instance, `Allow: /blog/*.md` could permit AI access to all markdown files in your blog. This allows for flexible and efficient content control over dynamically generated or structured paths within your Astro site.
- How often should I review and update my llms.txt for an Astro site?
- You should review and update your `llms.txt` file whenever there are significant changes to your Astro site's content, structure, or AI visibility strategy. This includes adding new content sections, removing old ones, or redefining how certain types of information should be handled by AI. Regular audits ensure ongoing compliance and effectiveness.
Explore further
Connected guides to keep going — short reads, all internally linked.