AI search readiness check
AI crawlers receive the same page as everyone else
The page returns the same status and substantially the same content to AI crawler user agents as to a default client.
high severityai.bot-fetch-parity
Run this check on your site
Runs just this check in quick mode. The report opens with the evidence and the fix.
Why it matters
Firewalls, bot management and CDN rules often block AI crawlers before robots.txt is even read. The site then looks open to humans and invisible to assistants.
How to fix it
Review the WAF, CDN and rate limiting rules that match the AI crawler user agents and allow them through.
The details
- Category
- AI search readiness
- Severity of a failure
- high
- Confidence
- high
- Needs a browser
- No
- Check id
- ai.bot-fetch-parity
References
Free tools that run this check
AI Crawler CheckerFind out whether ChatGPT, Claude, Perplexity and other AI assistants can read your site. The checker reads your robots.txt rules for each AI crawler and requests the page with each crawler's user agent, so a firewall block shows up too.AI Search Readiness CheckerGenerative engine optimization starts with access. Check that AI assistants can crawl your pages, receive the real content rather than a block page, and read it without running JavaScript.