How accurate is the parser?

It groups lines under User-agent blocks in order—good for quick reviews, not a substitute for crawler simulators or host-specific behavior.

Does blocking GPTBot remove my site from Google?

No. robots.txt rules are for crawlers that honor them. Search rankings depend on many factors beyond AI training crawlers.

Can I fetch my live robots.txt?

Yes—enter your public site URL and we request /robots.txt only.

Free toolsAI crawler rules checker

AI crawler & bot rules checker

See how your robots.txt lines up with common AI training crawlers—without hype, just the rules text.

Load robots.txt

Optional: fetch /robots.txt from a public site, then review how common AI-related crawlers are addressed.

Site URL

robots.txt

Paste or edit rules. The table below uses a simple parser: it is not a substitute for testing in your host environment.

AI-related crawlers (heuristic)

We match named User-agent blocks or the * wildcard when no named block exists. Blocking rules affect visibility of training crawlers—not your rankings in normal search.

OpenAI GPTBot1 Disallowagents: GPTBot
ChatGPT-User1 Allowagents: *
Google-Extended (Gemini training)1 Allowagents: *
Googlebot1 Allowagents: *
Common Crawl (CCBot)1 Allowagents: *
Anthropic1 Allowagents: *
ClaudeBot1 Allowagents: *
Bytespider1 Allowagents: *
Perplexity1 Allowagents: *
Applebot-Extended1 Allowagents: *

Frequently asked questions

How accurate is the parser?: It groups lines under User-agent blocks in order—good for quick reviews, not a substitute for crawler simulators or host-specific behavior.
Does blocking GPTBot remove my site from Google?: No. robots.txt rules are for crawlers that honor them. Search rankings depend on many factors beyond AI training crawlers.
Can I fetch my live robots.txt?: Yes—enter your public site URL and we request /robots.txt only.

AI crawler & bot rules checker

Frequently asked questions

Start learning in minutes