Free Tool · Robots.txt Tester
See exactly what your robots.txt blocks.
Paste a URL and test it against Googlebot and 8 AI crawlers, read every group, and catch the stray Disallow that hides your site. No signup to try it.
Try
No signupUnlimited & freeGooglebot + 8 AI bots
What the tester reads
01 / Access
Crawler access.
Test any path against Googlebot and the 8 AI crawlers — allowed or disallowed, with the exact rule that decides it.
02 / Groups
User-agent groups.
Every user-agent block and its allow/disallow rules, parsed and laid out so you can read your own file.
03 / Sitemap
Sitemap directives.
Whether robots.txt points crawlers at your sitemap — the pointer most sites forget to add.
04 / AI bots
AI crawler verdict.
GPTBot, ClaudeBot, PerplexityBot and more — blocked bots cannot cite you in AI answers.
05 / Raw
Raw file.
The exact robots.txt as served, so you can spot the stray Disallow that locks everyone out.
06 / Score
One score.
Access + discoverability weighted into a single 0–100 robots health score.