AI search readiness check
AI crawlers are allowed in robots.txt
What robots.txt says about each known AI crawler, classified as training, search or user triggered.
medium severityai.crawler-access
Run this check on your site
Runs just this check in quick mode. The report opens with the evidence and the fix.
Why it matters
Robots.txt is the only place a site states its policy towards AI crawlers. Blocking an AI search crawler removes the site from that assistant, blocking a training crawler is a legitimate choice.
How to fix it
Decide per crawler class. Allowing training crawlers while blocking search crawlers is a valid combination, just make it explicit in robots.txt.
The details
- Category
- AI search readiness
- Severity of a failure
- medium
- Confidence
- high
- Needs a browser
- No
- Check id
- ai.crawler-access
References
Free tools that run this check
Robots.txt CheckerCheck that robots.txt is published, parses cleanly and does not block the pages you want found. See at a glance which search engines and AI crawlers the rules allow.AI Crawler CheckerFind out whether ChatGPT, Claude, Perplexity and other AI assistants can read your site. The checker reads your robots.txt rules for each AI crawler and requests the page with each crawler's user agent, so a firewall block shows up too.AI Search Readiness CheckerGenerative engine optimization starts with access. Check that AI assistants can crawl your pages, receive the real content rather than a block page, and read it without running JavaScript.