Robots.txt Checker

What the robots.txt checker tells you

Robots.txt is the first file a crawler reads on your site. It says which paths each user agent may fetch, and it can point to your sitemap. The checker downloads that file, lists every Disallow and Allow rule per agent, and tests a URL you type against the rules for Googlebot, Bingbot, GPTBot and the other AI crawlers.

A rule that looks harmless in the file can block a whole section. "Disallow: /blog" also blocks "/blog-post" and "/blogs", because the match is a prefix, not a folder.

How to read the result

• Allowed means the crawler may fetch the URL. It does not mean the page is indexed; that depends on the meta robots tag and the canonical.
• Blocked by robots.txt is different from noindex. A blocked page can still appear in results as a bare URL if other sites link to it. To remove a page, let it be crawled and add noindex.
• No sitemap line is not an error, but adding "Sitemap: https://yoursite.com/sitemap.xml" is one line and saves Google a guess.
• GPTBot, ClaudeBot and PerplexityBot decide whether AI answers can cite you. Blocking them removes you from those answers, it does not protect the content, which is public anyway.

Common robots.txt mistakes

• A staging "Disallow: /" that shipped to production. The checker shows it in the first line.
• Blocking CSS and JavaScript folders. Google renders pages; without those files it sees a broken layout and ranks it that way.
• Blocking a folder to hide thin pages instead of fixing or noindexing them.
• Different rules on the www and non-www hosts. Each host serves its own robots.txt; check both.