Free ai search readiness check

AI Crawler Checker: is your site blocking ChatGPT, Claude or Perplexity?

Find out whether ChatGPT, Claude, Perplexity and other AI assistants can read your site. The checker reads your robots.txt rules for each AI crawler and requests the page with each crawler's user agent, so a firewall block shows up too.

Check AI crawler access

Enter a page. Its robots.txt rules are read and the page is requested as each AI crawler.

Checks in this tool:ai.crawler-access ai.bot-fetch-parity crawl.robots-txt

Why use the AI Crawler Checker

Two layers tested

robots.txt is only half the story. Firewalls and bot protection often block AI crawlers before robots.txt is read. Both are checked.

Grouped by purpose

Training, AI search and user request crawlers are shown separately, because blocking each one means something different.

The deciding rule

For every crawler the checker quotes the robots.txt rule that decides its access.

Same page check

The response to each AI user agent is compared with the response to a normal browser, so a block page or an empty shell is caught.

When to use the AI Crawler Checker

Missing from ChatGPT answers

Rule out a block when your site never appears in ChatGPT search or Perplexity.

After enabling bot protection

Cloudflare, Akamai and other WAFs added AI bot blocking. See what it does to your site.

Setting a policy

Decide which crawlers to allow with a clear view of what each one is for.

Publisher agreements

Verify that your robots.txt matches the policy you agreed to.

Agency audits

Show clients exactly which assistants can read their site, with evidence.

After CMS updates

Some platforms changed their default robots.txt to block AI crawlers. Check yours.

Understanding AI Crawler

Three kinds of AI crawlers

1Training. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and Bytespider collect pages to train models. Google-Extended and Applebot-Extended are robots.txt tokens that control training use by Google and Apple.
2AI search. OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so that assistants can find and cite them.
3User requests. ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked an assistant about it.
4Different stakes. Blocking training is a business decision. Blocking search and user crawlers keeps your pages out of those assistants' answers.

Why robots.txt is not enough

Many sites allow every crawler in robots.txt but serve AI user agents a 403 or a challenge page from their firewall. The crawler never reaches robots.txt, so the policy in the file does not matter. The checker requests the page as each crawler to catch this.

Some providers verify AI crawlers by their published IP ranges. A request from Aback servers with an AI user agent may be treated differently from the real crawler, so a block found here is a strong signal to check your firewall rules, not final proof.

What to do with the result

If you want to appear in AI answers, allow the search and user crawlers in robots.txt and in your bot management settings. If you do not want your content used for training, block the training crawlers and the Google-Extended and Applebot-Extended tokens. The two choices are independent.

Privacy, safety and limits

The check runs on Aback servers. It identifies itself as AbackAuditBot, and when it tests what an AI crawler receives it sends that crawler's user agent from the same servers. It never sends a cookie or a password and never changes the site.

It cannot tell you whether an assistant will cite you. It tells you whether the assistant can reach and read your pages, which is the part you control.

Related tools and guides

Robots.txt CheckerCheck that robots.txt is published, parses cleanly and does not block the pages you want found. See at a glance which search engines and AI crawlers the rules allow.AI Search Readiness CheckerGenerative engine optimization starts with access. Check that AI assistants can crawl your pages, receive the real content rather than a block page, and read it without running JavaScript.llms.txt Validatorllms.txt is a Markdown file that gives language models a short, curated map of a site. Check that yours is published, follows the proposed format and links to pages that work.Robots.txt GeneratorWrite a correct robots.txt in a minute. Pick a policy for AI crawlers, list the paths to keep crawlers out of, add your sitemap and copy the file.JavaScript SEO CheckerMany crawlers, including most AI crawlers, read the HTML the server sends and never run your JavaScript. Check how much of the page they actually see.SEO CheckerFind the technical problems that keep a page out of search results. The checker reads the page the way a search crawler does and lists what to fix first, with the evidence behind every verdict.Guide: robots.txt for AI crawlersWhich AI crawlers exist, what each one does, and how to write robots.txt rules that block model training while keeping your site visible in ChatGPT, Claude and Perplexity answers.Guide: AI search readinessThe technical side of generative engine optimization. How ChatGPT, Claude, Perplexity and Google AI features find and read pages, and the checks that make sure they can read yours.

Frequently asked questions

Is my site blocking ChatGPT?

Check the rows for GPTBot, OAI-SearchBot and ChatGPT-User. GPTBot is for training only. OAI-SearchBot decides whether you appear in ChatGPT search, and ChatGPT-User decides whether ChatGPT can open your page when someone asks about it.

Should I block AI crawlers?

That is your decision. Blocking training crawlers keeps your content out of future models and costs nothing in search. Blocking search and user crawlers keeps you out of the answers of those assistants.

Does blocking Google-Extended remove me from Google?

No. Google-Extended only controls whether your content is used to train and ground Gemini. Google Search crawling is controlled by Googlebot, which is unaffected.

Why do I see a block even though robots.txt allows everything?

Your firewall, CDN or bot protection is blocking the user agent before robots.txt matters. Look for AI bot rules in Cloudflare, your WAF or your hosting control panel.

Can the checker pretend to be the real GPTBot?

No, and it should not. It sends the user agent from Aback servers. Providers that verify crawlers by IP may answer it differently from the real crawler, which is why the result is labelled with its confidence.

Which crawlers are checked?

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, Meta-ExternalAgent and Bytespider, plus the Google-Extended and Applebot-Extended tokens in robots.txt. The list is maintained at /api/v1/ai-bots.

Does allowing AI crawlers mean I will be cited?

No. Access is a precondition, not a guarantee. Assistants decide what to cite based on relevance and quality, which no checker can measure.

How do I allow AI search but block training?

Add a group for each training crawler with Disallow: / and leave the search and user crawlers allowed. The robots.txt generator has a preset for exactly this.