Robots.txt and XML Sitemaps: The Complete SEO Guide
Your robots.txt file and XML sitemap work together as the traffic controller and roadmap for search engine and AI crawlers. Here is how to configure both without accidentally blocking your site.
Use robots.txt to keep crawlers out of private admin, cart and internal search paths while leaving public pages, CSS and JS open to Googlebot. Manage AI training bots (GPTBot, ClaudeBot, Google-Extended) with dedicated user-agent groups, and list only 200 OK, indexable, self-canonical URLs in your sitemap.xml.
Convert Your Files Right Now
No sign-up required, zero file uploads. 100% free and private in your browser.
How Googlebot and AI crawlers read robots.txt (RFC 9309)
Under RFC 9309, a crawler matches only the most specific User-agent group (so Googlebot ignores User-agent: * if a User-agent: Googlebot group exists). Within that group, the rule matching the longest path prefix wins, and Allow beats Disallow on ties. Build clean files with platform presets and AI bot toggles in the Robots.txt Generator, and test specific URLs line by line in the Robots.txt Tester.
Building and validating a clean XML sitemap
Your sitemap.xml should contain only canonical, indexable URLs that return HTTP 200. Including 301 redirects, 404 pages, or noindex URLs wastes crawl budget and triggers Search Console warnings. Generate a filtered sitemap with the XML Sitemap Generator and check existing sitemaps or sitemap indexes for syntax, namespace and live HTTP status errors in the Sitemap Validator.
Try Free Web Tools Mentioned in This Guide
Robots.txt Tester
Test whether Googlebot, Bingbot and AI crawlers can crawl any URL — and find robots.txt mistakes.
Robots.txt Generator
Build a clean, RFC 9309 robots.txt with presets for Next.js, WordPress & Shopify, plus 1-click AI crawler controls.
XML Sitemap Generator
Crawl your site and generate a clean, valid sitemap.xml — only indexable, canonical URLs.
Sitemap Validator
Validate any sitemap.xml: syntax, namespaces, absolute URLs, lastmod dates, limits and live URL status.
Canonical URL & Indexability Checker
Inspect <link rel="canonical">, meta robots, X-Robots-Tag and HTTP status for a single page or up to 10 URLs in bulk.
Website Technical SEO Audit
All-in-one technical SEO suite: full website crawl audit, broken link checker (404s), and canonical & indexability analyzer.
Frequently Asked Questions
Does blocking Google-Extended affect Google Search rankings?↓
No. Google-Extended only controls whether your content is used to train Gemini generative models; Googlebot controls Google Search indexing.
Should I block noindex pages in robots.txt?↓
No. If robots.txt blocks a URL, Googlebot cannot crawl the page to read the noindex tag and may still index the URL from external links.