Skip to content
FlexibleToolsAI
SEO Toolkit · Free, No Sign-Up

Robots.txt Tester — Check Googlebot & AI Crawler Access

Fetch any site's robots.txt (or paste your own), then test URLs against Googlebot, Bingbot, GPTBot and any other crawler. The tester applies Google's exact matching rules — most specific user-agent group, longest matching path, Allow wins ties — and flags mistakes line by line.

Enter your domain URL

Enter any website domain or URL to inspect its live robots.txt file, test crawler access directives, and find indexing issues.

Try popular:

Google-Accurate

Pixel limits & real search data

Instant Results

Full analysis in seconds

No Sign-Up

Free, no credits or email

What Does Robots.txt Tester Do & How Does It Work?

Enter your domain to load its robots.txt, or paste the file. Add the URLs you want to test and choose a crawler. The tester shows whether each URL is allowed or blocked and which line decided it, using the same precedence rules Google documents (RFC 9309), plus a table of which search and AI crawlers can reach your homepage.

How to Use Robots.txt Tester (Step-by-Step)

  1. 1Enter your domain and click Fetch robots.txt — or paste the file's contents.
  2. 2List the URLs or paths you want to test, one per line.
  3. 3Choose a crawler such as Googlebot, Bingbot or GPTBot (or type any user-agent).
  4. 4Read each result and the rule that matched, then edit the file to try fixes before publishing.

How Google reads robots.txt

A crawler follows only the group whose User-agent line matches it most specifically — Googlebot ignores the * group entirely if there's a Googlebot group. Inside that group the rule with the longest matching path wins, and if an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ marks the end of the URL.

Google ignores crawl-delay and noindex lines, reads only the first 500 KiB, and treats a 5xx error on robots.txt as “don't crawl the site” until it can fetch it again. A 404 means “crawl everything”.

Blocking crawling is not blocking indexing

Disallow stops crawling, not indexing: a blocked URL can still appear in results (without a description) if other pages link to it. To keep a page out of Google, allow crawling and add a noindex robots meta tag — Google has to crawl the page to see it. Check a page's indexing signals with the Meta Tag Analyzer.

AI crawlers are controlled the same way. GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot each have their own user-agent; Google-Extended controls whether Google may use your content for its AI models without affecting Search.

Recommended Page Formats & Best Practice Guide

Cheat-sheet showing how Google evaluates this data and which page format ranks best.

Rule Precedence (RFC 9309)How Googlebot Evaluates ItCommon Mistake
1. Most Specific User-AgentA Googlebot group completely overrides the wildcard * groupExpecting Googlebot to inherit * rules when a Googlebot block exists
2. Longest Matching PathThe rule matching the most characters in the URL path winsAssuming top-to-bottom file order decides whether a URL is blocked
3. Allow Wins TiesIf Allow and Disallow match the exact same path length, Allow winsUsing robots.txt Disallow to deindex a page instead of a noindex meta tag
Actionable Takeaway: Aligning your target page layout with these standards helps satisfy Google's Quality Rater Guidelines and keeps search bounce rates low.

Input & output compatibility

Accepts

  • Live domain (auto-fetches /robots.txt) or pasted rules
  • Test URLs + crawler selection (Googlebot, Bingbot, GPTBot, ClaudeBot)

Produces

  • Allowed / Blocked verdict with exact winning line number
  • Search & AI crawler access matrix

Frequently Asked Questions

Is this the same as Google's robots.txt tester?

Google retired its old robots.txt Tester in 2023 and now shows a robots.txt report in Search Console. This tool applies the same documented matching rules, lets you test any URL and crawler, and lets you edit the file to try changes before you publish them.

Why is my page still in Google if robots.txt blocks it?

robots.txt controls crawling, not indexing. Google can index a blocked URL from links alone. Remove the Disallow and add a noindex meta tag instead, then let Google recrawl it.

How do I block AI crawlers like GPTBot?

Add a group for each one, for example “User-agent: GPTBot” followed by “Disallow: /”. Use the crawler table here to confirm which bots are blocked after your change.

Does Google support crawl-delay?

No. Googlebot ignores crawl-delay. Bing and Yandex honour it. Google adjusts its crawl rate automatically based on how your server responds.

Is this SEO tool free? Do I need to sign up?

It's completely free with no sign-up, no email and no daily credits. Use it as often as you like.

Guides & Tutorials

Explore More Free Browser Tools

View All 290+ Tools Directory →

All tools run client-side in your browser with zero data uploads, no sign-up, and zero tracking. Discover other popular converters and calculators:

Search Standards & Diagnostic Methodology

Robots.txt Tester is engineered across modern search standards — combining official Google Search Central guidelines with semantic SEO, Answer Engine snippet formatting, and AI citation readiness:

  • 1. Accurate On-Page & Technical Diagnostics: Evaluates SERP pixel budgets (~580px titles, ~920px descriptions), canonical consistency, heading hierarchies, and crawl directives to prevent indexing drops.
  • 2. Featured Snippet & Voice Search Optimization: Structures content for 40–55 word direct-answer extraction, conversational PAA query matching, and valid FAQPage/HowTo JSON-LD schema.
  • 3. Generative AI Citations & Entity Signals: Generates entity-connected Schema.org @graph markups and semantic LSI terms so ChatGPT, Perplexity, and Claude cite your pages.
  • 4. Google AI Overview Alignment: Formats answers for direct indexing in Google AI Overviews while maintaining clear crawler governance for AI bots.
  • 5. Search Experience & Zero Friction: 100% free in-browser execution with no credit card or account wall, fast Core Web Vitals performance, and 1-click output copying.

Suggest a Feature or Improvement for Robots.txt Tester

Need custom options, higher limits, or extra format support? Let our engineering team know!