Robots.txt Tester
Fetch any site's robots.txt (or paste your own), then test URLs against Googlebot, Bingbot, GPTBot and any other crawler. The tester applies Google's exact matching rules — most specific user-agent group, longest matching path, Allow wins ties — and flags mistakes line by line.
Enter your domain to load its robots.txt, or paste the file. Add the URLs you want to test and choose a crawler. The tester shows whether each URL is allowed or blocked and which line decided it, using the same precedence rules Google documents (RFC 9309), plus a table of which search and AI crawlers can reach your homepage.
1. Load or paste robots.txt
2. Test URLs
Pick a crawler or type any user-agent token.
- Allowed/No matching rule — allowed by default
- Blocked/admin/settingsLine 2:
Disallow: /admin/ - Allowed/admin/public/helpLine 3:
Allow: /admin/public/ - Allowed/blog/my-postNo matching rule — allowed by default
Group used for Googlebot: *
Who can crawl your homepage?
GPTBot, ClaudeBot and PerplexityBot are AI crawlers; Google-Extended controls use of your content for Google's AI models (it doesn't affect Search).
Issues (0)
No problems found.
Summary
- https://example.com/sitemap.xml
Google-Accurate
Pixel limits & real search data
Instant Results
Full analysis in seconds
No Sign-Up
Free, no credits or email
Suggest a Feature or Improvement for Robots.txt Tester
Need custom options, higher limits, or extra format support? Let our engineering team know!
How to use Robots.txt Tester
- 1Enter your domain and click Fetch robots.txt — or paste the file's contents.
- 2List the URLs or paths you want to test, one per line.
- 3Choose a crawler such as Googlebot, Bingbot or GPTBot (or type any user-agent).
- 4Read each result and the rule that matched, then edit the file to try fixes before publishing.
How Google reads robots.txt
A crawler follows only the group whose User-agent line matches it most specifically — Googlebot ignores the * group entirely if there's a Googlebot group. Inside that group the rule with the longest matching path wins, and if an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ marks the end of the URL.
Google ignores crawl-delay and noindex lines, reads only the first 500 KiB, and treats a 5xx error on robots.txt as “don't crawl the site” until it can fetch it again. A 404 means “crawl everything”.
Blocking crawling is not blocking indexing
Disallow stops crawling, not indexing: a blocked URL can still appear in results (without a description) if other pages link to it. To keep a page out of Google, allow crawling and add a noindex robots meta tag — Google has to crawl the page to see it. Check a page's indexing signals with the Meta Tag Analyzer.
AI crawlers are controlled the same way. GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot each have their own user-agent; Google-Extended controls whether Google may use your content for its AI models without affecting Search.
Frequently Asked Questions
Is this the same as Google's robots.txt tester?
Google retired its old robots.txt Tester in 2023 and now shows a robots.txt report in Search Console. This tool applies the same documented matching rules, lets you test any URL and crawler, and lets you edit the file to try changes before you publish them.
Why is my page still in Google if robots.txt blocks it?
robots.txt controls crawling, not indexing. Google can index a blocked URL from links alone. Remove the Disallow and add a noindex meta tag instead, then let Google recrawl it.
How do I block AI crawlers like GPTBot?
Add a group for each one, for example “User-agent: GPTBot” followed by “Disallow: /”. Use the crawler table here to confirm which bots are blocked after your change.
Does Google support crawl-delay?
No. Googlebot ignores crawl-delay. Bing and Yandex honour it. Google adjusts its crawl rate automatically based on how your server responds.
Is this SEO tool free? Do I need to sign up?
It's completely free with no sign-up, no email and no daily credits. Use it as often as you like.
Related Tools
XML Sitemap Generator
Crawl your site and generate a clean, valid sitemap.xml — only indexable, canonical URLs.
Sitemap Validator
Validate any sitemap.xml: syntax, namespaces, absolute URLs, lastmod dates, limits and live URL status.
Meta Tag Analyzer
See and check every meta tag on a page — title, description, robots, canonical, Open Graph and Twitter.