What should an AI crawler access audit test?
A complete access audit checks four layers in order: declared permission in robots.txt, network access through the CDN or firewall, the final HTTP response after redirects, and the useful HTML available without browser-side JavaScript. Passing one layer does not prove the next one works.
| Layer | Pass condition | Common failure |
|---|---|---|
| robots.txt | The relevant user agent is not disallowed | A wildcard rule overrides an intended allow |
| Network edge | The bot receives the same reachable origin as a visitor | Bot protection serves 403, 429, or a challenge |
| HTTP | One canonical 200 response after sensible redirects | Redirect loop, soft 404, or locale trap |
| Rendered content | Title, answer, links, and structured data exist in initial HTML | An empty app shell depends on client JavaScript |
Which crawler names need separate checks?
Do not treat every AI user agent as interchangeable. Search retrieval, user-triggered fetches, and model training can use different crawlers and different controls. Decide your policy by purpose, then test the exact official user-agent token documented by each provider.
- OpenAI: test OAI-SearchBot for search discovery, ChatGPT-User for user-triggered retrieval, and GPTBot according to your training policy.
- Anthropic: review ClaudeBot and any current documented retrieval agents separately.
- Perplexity: test PerplexityBot and user-triggered retrieval according to current documentation.
- Google: distinguish Googlebot search access from Google-Extended controls for generative training and grounding.
Crawler names and policies change. Use each provider's current documentation as the authority, and date your audit notes.
How do you run a repeatable crawler test?
- Choose five representative URLs, including one deep page that is not linked from the homepage.
- Fetch robots.txt and record the rule that applies to each user agent.
- Request each URL with the official user-agent string and record status, final URL, content type, response size, and response time.
- Inspect the returned HTML for the canonical, H1, direct answer, main links, and JSON-LD.
- Compare the bot response with a normal browser response and investigate meaningful differences.
- Check edge and origin logs for challenges, rate limits, and repeated failed requests.
curl -L -A 'OAI-SearchBot' -D headers.txt https://example.com/ -o page.html grep -Ei '<title|<h1|application/ld\+json|rel="canonical"' page.html
How do you diagnose a failed crawler request?
| Observed result | Likely cause | Next check |
|---|---|---|
| 403 or challenge page | CDN, WAF, or bot-management rule | Inspect the matching edge event and rule ID |
| 429 | Rate limit shared across automated traffic | Review bot-specific limits and retry guidance |
| 200 with tiny HTML | Client-only rendering or blocked data request | View the initial source and server logs |
| Redirect to login or locale picker | Middleware or geolocation rule | Test cookies, Accept-Language, and canonical behavior |
| Correct page but no schema | Structured data injected only in the browser | Move critical markup into server-rendered HTML |
What proves the access fix worked?
A fix is verified only when the affected user agent receives a canonical 200 page with the same substantive answer a visitor sees. Save the command, timestamp, headers, and a hash or excerpt of the body. Then confirm a real visit in server logs; synthetic access proves possibility, while logs prove actual crawling.
- Retest every affected template, not only the homepage.
- Confirm robots.txt and sitemap references remain consistent.
- Check that caching does not serve a stale blocked response.
- Add the audit to release checks after CDN, firewall, or rendering changes.