Free Robots.txt Tester & Validator

Validate your robots.txt for crawl issues.

See whether Google and AI crawlers like GPTBot and ClaudeBot can read your site:

About this robots.txt validator & checker

Our free robots.txt tester fetches and validates your robots.txt file in real time. Enter your domain, the robots.txt tester pulls the file, parses every directive, surfaces any conflicts with Googlebot rules and discovers any linked XML sitemaps. Use it as a robots.txt validator before launches, after migrations, or any time you suspect Google can't crawl a page it should. Built on the same parsing logic Google's own Search Console uses internally.

How to use Robots.txt Validator & Checker

  1. 1

    Enter Domain

    Type in your website's URL to automatically locate the robots.txt file.

  2. 2

    Automated Fetch

    Our system fetches and analyzes your configuration file in real-time.

  3. 3

    Sitemap Discovery

    We automatically detect and display the content of any linked XML sitemaps.

  4. 4

    Validation Results

    Review highlights for crawlability, sitemap presence, and admin protection.

More free SEO tools

“I am absolutely obsessed with SEO. I spend my days mastering the backlinks and growth strategies you need to scale on auto-pilot.”
TiboTiboFounder of Outrank, SuperX
Outrank

Grow organic traffic on auto‑pilot

Outrank writes and publishes SEO articles and builds backlinks for you, every day.

Try Outrank free

Frequently asked questions

What if my site doesn't have a robots.txt?

Our tool will notify you. While it's not strictly required, having one is best practice for controlling how search engines crawl your site and saving your crawl budget.

Can I edit the file here?

Currently, we provide validation and sitemap viewing. You'll need to update the file on your server (usually via FTP or your CMS) once you identify issues.

What is crawl budget?

Crawl budget is the number of pages a search engine crawler will visit on your site in a given timeframe. Robots.txt helps prioritize the most important pages.

Does robots.txt hide pages from users?

No. Robots.txt only provides instructions to crawlers. If a page is blocked via robots.txt, it can still be accessed directly by a user if they have the link.

Is robots.txt a security feature?

Not really. It shouldn't be used to hide sensitive data, as the file itself is public. For security, use password protection or 'noindex' tags instead.

Can I test if a specific page is blocked?

Yes. Use Can this bot crawl a page, pick a crawler and type a path. The tool shows whether it is allowed and which rule made the decision.

How do you decide if a path is allowed?

We follow Google's rules: the most specific user-agent group applies, the longest matching rule wins, and Allow wins a tie. Wildcards (*) and end anchors ($) are supported.

Should I block AI crawlers like GPTBot?

It depends on your goals. Blocking them keeps your content out of AI training and answers, but allowing them can get your brand mentioned and cited in AI search results.

What is Google-Extended?

A token Google uses to control whether your content helps train Gemini models. Blocking it does not affect Google Search rankings.

Where should robots.txt live?

At the root of your domain, for example example.com/robots.txt. Crawlers do not look for it anywhere else.

How do I disallow all bots in robots.txt?

Add two lines: User-agent: * and then Disallow: /. That asks every crawler to skip the whole site. Use it only on staging or private sites, because it also keeps your pages out of Google.