IndexMon
Start free

Free tool

Robots.txt checker

Fetch a site's robots.txt and test any path against Googlebot.

How the test works

  • Picks the most specific user-agent group — googlebot beats *.
  • The longest matching rule wins. On a tie, Allow wins.
  • Supports * wildcards and $ end anchors.
  • A robots.txt that returns 5xx is treated as "block everything".

Want to see all 9 crawlers (including GPTBot, ClaudeBot and PerplexityBot) at once? Run the deep check.

More free tools

Index checkerEvery reason a URL can't be indexed→Bulk index checker25 URLs, pass/fail, CSV→Google index checkerSee a page the way Googlebot does→Website index checkerSite-wide blockers on your homepage→Noindex checkerMeta tags and X-Robots-Tag headers→

FAQ

Questions

Does Disallow remove a page from Google?+

No. It stops crawling. To remove a page, allow crawling and add noindex.

Where does robots.txt live?+

Always at the root: https://yourdomain.com/robots.txt