Robots.txt Checker and Tester

What robots.txt does

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. It is a request, not a lock: well behaved crawlers follow it, and anything private still needs authentication.

Blocking a page in robots.txt does not remove it from search results. A blocked url can still be indexed from links, without a description. To keep a page out of the index, allow crawling and use a noindex directive instead.

How the rules are matched

1. 1Pick the group. A crawler uses the group that names its product token. Only if no group names it does it fall back to User-agent: *.
2. 2Find the rules. Inside the group, every Allow and Disallow whose path matches the url is a candidate. * matches any characters and $ anchors the end.
3. 3Longest wins. The rule with the longest path wins. On a tie, Allow wins.
4. 4No rule means allowed. A url that matches nothing may be crawled.

Training crawlers and search crawlers

AI companies run different crawlers for different jobs. GPTBot and ClaudeBot collect content to train models. OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so that an assistant can cite them. ChatGPT-User and similar agents fetch a page because a person asked about it.

Blocking training is a legitimate policy choice, so the checker reports it as information. Blocking search and user agents means your pages cannot appear in those assistants, so it is reported as a warning. The decision is yours.

Privacy, safety and limits

The check runs on Aback servers as AbackAuditBot. It sends ordinary GET and HEAD requests, never a cookie, a password or a form, and it never changes anything on the site.

The result is stored as a report that anyone with its link can open, and it is deleted after 30 days. Do not check addresses that carry secrets in the path or the query string.