Free toolsAI crawler rules checker
AI crawler & bot rules checker
See how your robots.txt lines up with common AI training crawlers—without hype, just the rules text.
Load robots.txt
Optional: fetch
/robots.txt from a public site, then review how common AI-related crawlers are addressed.robots.txt
Paste or edit rules. The table below uses a simple parser: it is not a substitute for testing in your host environment.
AI-related crawlers (heuristic)
We match named
User-agent blocks or the * wildcard when no named block exists. Blocking rules affect visibility of training crawlers—not your rankings in normal search.- OpenAI GPTBot1 Disallowagents: GPTBot
- ChatGPT-User1 Allowagents: *
- Google-Extended (Gemini training)1 Allowagents: *
- Googlebot1 Allowagents: *
- Common Crawl (CCBot)1 Allowagents: *
- Anthropic1 Allowagents: *
- ClaudeBot1 Allowagents: *
- Bytespider1 Allowagents: *
- Perplexity1 Allowagents: *
- Applebot-Extended1 Allowagents: *
Frequently asked questions
- How accurate is the parser?
- It groups lines under User-agent blocks in order—good for quick reviews, not a substitute for crawler simulators or host-specific behavior.
- Does blocking GPTBot remove my site from Google?
- No. robots.txt rules are for crawlers that honor them. Search rankings depend on many factors beyond AI training crawlers.
- Can I fetch my live robots.txt?
- Yes—enter your public site URL and we request /robots.txt only.
Learn anything.
Describe it.
Get a course.
Open the dashboard, say what you want to master, and let Ailurn draft the outline and lessons—then learn with tools that stay in context.