SaaS: How to audit AI crawler access
Audit AI crawler access by checking robots.txt, testing each official user agent against important URLs, comparing the returned status and HTML with a normal browser request, and reviewing CDN or firewall logs for blocks. A robots.txt Allow rule is not enough if the server returns a challenge, 403, empty shell, or different content.
The challenge
Test whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can fetch your site, diagnose blocks, and verify each fix safely.
Reusable implementation checklist
- Test retrieval crawlers separately from training crawlers because their roles and controls differ.
- Verify the final response body, not only robots.txt or the HTTP status.
- Audit representative page types: homepage, article, product, documentation, and a deep URL.
- Fix CDN and firewall rules before rewriting content; inaccessible pages cannot earn citations.
- Repeat the checks after deployments and monitor server logs for real crawler visits.
- Repeat the original prompt and analytics checks on the same schedule, documenting unrelated site changes.
Verify each change on your own site
| Signal | What to check | Free tool |
|---|---|---|
| robots.txt | The AI crawlers you intend to allow are not blocked by a broader rule. | Open |
| llms.txt | The file returns HTTP 200, uses Markdown headings and every link resolves. | Open |
| Structured data | JSON-LD parses without errors and matches what visitors see on the page. | Open |
| FAQ answers | Each answer is visible on the page and written as a complete sentence. | Open |
| Prompt tracking | The same prompts are re-run on a fixed schedule and results are logged. | Open |
What should an AI crawler access audit test?
A complete access audit checks four layers in order: declared permission in robots.txt, network access through the CDN or firewall, the final HTTP response after redirects, and the useful HTML available without browser-side JavaScript. Passing one layer does not prove the next one works.
| Layer | Pass condition | Common failure |
|---|---|---|
| robots.txt | The relevant user agent is not disallowed | A wildcard rule overrides an intended allow |
| Network edge | The bot receives the same reachable origin as a visitor | Bot protection serves 403, 429, or a challenge |
| HTTP | One canonical 200 response after sensible redirects | Redirect loop, soft 404, or locale trap |
| Rendered content | Title, answer, links, and structured data exist in initial HTML | An empty app shell depends on client JavaScript |
Which crawler names need separate checks?
Do not treat every AI user agent as interchangeable. Search retrieval, user-triggered fetches, and model training can use different crawlers and different controls. Decide your policy by purpose, then test the exact official user-agent token documented by each provider.
- OpenAI: test OAI-SearchBot for search discovery, ChatGPT-User for user-triggered retrieval, and GPTBot according to your training policy.
- Anthropic: review ClaudeBot and any current documented retrieval agents separately.
- Perplexity: test PerplexityBot and user-triggered retrieval according to current documentation.
- Google: distinguish Googlebot search access from Google-Extended controls for generative training and grounding.
How do you run a repeatable crawler test?
- Choose five representative URLs, including one deep page that is not linked from the homepage.
- Fetch robots.txt and record the rule that applies to each user agent.
- Request each URL with the official user-agent string and record status, final URL, content type, response size, and response time.
- Inspect the returned HTML for the canonical, H1, direct answer, main links, and JSON-LD.
- Compare the bot response with a normal browser response and investigate meaningful differences.
- Check edge and origin logs for challenges, rate limits, and repeated failed requests.
curl -L -A 'OAI-SearchBot' -D headers.txt https://example.com/ -o page.html
grep -Ei '<title|<h1|application/ld\+json|rel="canonical"' page.htmlHow do you diagnose a failed crawler request?
| Observed result | Likely cause | Next check |
|---|---|---|
| 403 or challenge page | CDN, WAF, or bot-management rule | Inspect the matching edge event and rule ID |
| 429 | Rate limit shared across automated traffic | Review bot-specific limits and retry guidance |
| 200 with tiny HTML | Client-only rendering or blocked data request | View the initial source and server logs |
| Redirect to login or locale picker | Middleware or geolocation rule | Test cookies, Accept-Language, and canonical behavior |
| Correct page but no schema | Structured data injected only in the browser | Move critical markup into server-rendered HTML |
What proves the access fix worked?
A fix is verified only when the affected user agent receives a canonical 200 page with the same substantive answer a visitor sees. Save the command, timestamp, headers, and a hash or excerpt of the body. Then confirm a real visit in server logs; synthetic access proves possibility, while logs prove actual crawling.
- Retest every affected template, not only the homepage.
- Confirm robots.txt and sitemap references remain consistent.
- Check that caching does not serve a stale blocked response.
- Add the audit to release checks after CDN, firewall, or rendering changes.
Implementation steps
- 1Choose representative pages
Select important URLs across each template and include at least one deep page.
- 2Read the applicable robots rules
Evaluate each official AI user agent independently, including wildcard rules.
- 3Fetch and compare responses
Record redirects, status, headers, body size, visible content, and structured data.
- 4Trace any block
Use edge and origin logs to identify the exact rule or rendering dependency.
- 5Retest and monitor
Verify the corrected response and watch for genuine crawler visits over time.
Frequently asked questions
- Does allowing GPTBot also allow ChatGPT Search?
- Not necessarily. OpenAI documents separate agents for training, search discovery, and user-triggered retrieval. Audit and configure each current official agent according to the access you intend to grant.
- Why does an AI crawler get a 403 when robots.txt allows it?
- robots.txt declares crawl preference but does not bypass a CDN, firewall, authentication layer, or bot challenge. Inspect the edge event and origin logs to identify which layer returned the denial.
- Is a 200 response enough?
- No. A 200 can contain a challenge page, empty application shell, soft 404, or incomplete localized page. Inspect the returned HTML for the actual answer, canonical, links, and structured data.
- How often should crawler access be audited?
- Retest after changes to hosting, CDN rules, rendering, redirects, authentication, or robots.txt. For stable sites, a scheduled monthly check plus log monitoring catches most regressions.
- Should every AI crawler be allowed?
- That is a policy decision. Separate user-triggered retrieval and search discovery from model training, then allow or block each documented agent deliberately rather than using one blanket rule.
Primary references
Check your own starting point
Run a free OptimAIze scan, review the evidence and choose which changes to verify on your site.
Start your free scan