Key takeaways
- Training crawlers, search crawlers and user request agents are separate. Decide for each group.
- Blocking GPTBot does not remove you from ChatGPT search. OAI-SearchBot and ChatGPT-User decide that.
- Google-Extended and Applebot-Extended are control tokens, not crawlers. They do not affect Google Search or Apple search results.
- Your firewall can block AI crawlers before robots.txt is ever read. Check both.
On this page
The three kinds of AI crawlers
| Purpose | Crawlers | What blocking does |
|---|---|---|
| Model training | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, plus the Google-Extended and Applebot-Extended tokens | Your content is not used to train future models |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages are not indexed for those assistants’ search, so they are not shown or cited |
| User requests | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot open your page when a person asks it to |
The current list, with each crawler’s vendor and documentation, is published at /api/v1/ai-bots.
Choose a policy
There are three sensible policies. Pick the one that matches what you want.
Allow everything
The default when robots.txt says nothing about AI crawlers. Your content can be used for training and appears in AI answers. Many businesses that want the most visibility choose this.
Block training, allow search
Common for publishers and anyone whose content is the product. Training crawlers stay out, and assistants can still find, cite and link to your pages.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlSeveral User-agent lines in a row share the rules that follow them. That is valid under RFC 9309 and keeps the file short.
Block every AI crawler
Add the search and user agents to the group above. Your pages stop appearing in those assistants. Be sure this is what you want: people increasingly ask assistants about products and services, and a blocked site cannot be part of the answer.
How the rules are matched
- A crawler uses the group that names its token. Only if no group names it does it use
User-agent: *. So aGPTBotgroup withAllow: /overrides a wildcard block for GPTBot. - Inside a group, the longest matching path wins, and
Allowwins a tie. - Tokens are matched case insensitively.
Check the second layer: your firewall
Many sites have robots.txt that allows everything and a CDN that blocks AI user agents anyway. Bot management products added one click AI blocking, and some enabled it by default for new sites. A blocked crawler never reaches robots.txt, so the file does not matter.
The AI crawler checker requests your page with each crawler’s user agent and compares the answer with a normal browser’s. A 403 or a challenge page for OAI-SearchBot while a browser gets 200 means the firewall is deciding, not robots.txt.
Providers that verify crawlers by IP address may treat a test request differently from the real crawler. A block found by a checker is a strong reason to review your firewall settings, then confirm with your provider’s logs.
Write and test
- Write the file with the robots.txt generator. It lists every known crawler with its purpose.
- Publish it at the root of each host that serves pages.
- Test it with the robots.txt checker and the AI crawler checker.
- Revisit it twice a year. New crawlers appear, and providers rename them.
Check your own site
The AI Crawler Checker runs the checks from this guide on any public page, free and without an account.
Open the AI Crawler CheckerFrequently asked questions
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler. Google Search uses Googlebot, which is unaffected.
Do AI crawlers respect robots.txt?
The major providers say their named crawlers do, and publish their user agents and IP ranges. Some user request agents treat a fetch on behalf of a person differently, so read each provider's documentation.
Can I allow AI search but block training?
Yes. Disallow the training crawlers and the Google-Extended and Applebot-Extended tokens, and leave OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user agents allowed. The robots.txt generator has this as a preset.
Why do I see blocks in the AI crawler checker even though robots.txt allows everything?
Your CDN, firewall or bot management is answering those user agents with a block or a challenge page. Look for AI bot settings in Cloudflare, your WAF or your host.