Crawlability check
The default robots rules do not block the site
The wildcard rules in robots.txt do not block the audited URL or the key paths of the site.
critical severitycrawl.robots-blocking
Run this check on your site
Runs just this check in quick mode. The report opens with the evidence and the fix.
Why it matters
A blanket Disallow in the wildcard group hides the whole site from every crawler that does not name itself in robots.txt. This is the single most common reason a site cannot be indexed.
How to fix it
Remove the blocking rule, or scope it to the paths that really should be private. Test the file against the audited URL before publishing it.
The details
- Category
- Crawlability
- Severity of a failure
- critical
- Confidence
- high
- Needs a browser
- No
- Check id
- crawl.robots-blocking
References
Free tools that run this check
SEO CheckerFind the technical problems that keep a page out of search results. The checker reads the page the way a search crawler does and lists what to fix first, with the evidence behind every verdict.Robots.txt CheckerCheck that robots.txt is published, parses cleanly and does not block the pages you want found. See at a glance which search engines and AI crawlers the rules allow.Indexability CheckerA page can be blocked from the index in four places: the status code, a noindex directive, robots.txt and the canonical link. Check all four at once and see exactly which one is the problem.