Crawlability check
robots.txt exists and is usable
The file exists, returns 200, parses, and stays under 500 KiB.
high severitycrawl.robots-txt
Run this check on your site
Runs just this check in quick mode. The report opens with the evidence and the fix.
Why it matters
robots.txt is how a site tells crawlers where to go. A missing or broken file removes that control and hides real crawl problems.
How to fix it
Publish a plain text robots.txt at the root of the site and keep it small. Test it with the robots testing tool before deploying.
The details
- Category
- Crawlability
- Severity of a failure
- high
- Confidence
- high
- Needs a browser
- No
- Check id
- crawl.robots-txt
References
Free tools that run this check
SEO CheckerFind the technical problems that keep a page out of search results. The checker reads the page the way a search crawler does and lists what to fix first, with the evidence behind every verdict.Robots.txt CheckerCheck that robots.txt is published, parses cleanly and does not block the pages you want found. See at a glance which search engines and AI crawlers the rules allow.Sitemap CheckerCheck that your sitemap exists, parses as valid XML, is declared in robots.txt and lists urls that actually work. A broken sitemap wastes the crawl budget you give it.Indexability CheckerA page can be blocked from the index in four places: the status code, a noindex directive, robots.txt and the canonical link. Check all four at once and see exactly which one is the problem.