Does your robots.txt block AI crawlers?
Paste your real robots.txt below and check it against the real AI bots that matter: GPTBot, ClaudeBot, PerplexityBot, and the rest. Nothing is sent anywhere, this runs entirely in your browser.
- If an AI crawler can't read your page, it can't cite your page, no matter how good the content is.
- A single stray
Disallow: /under the wrong User-agent block can quietly block every one of them. - Paste your robots.txt below to check it against the real AI bots that actually matter.
Why this is a real problem, not a hypothetical
GapCite's own free scanner hits this directly: when a page's robots.txt blocks automated tools, the scan fails outright, with the real, plain error "That page's robots.txt file blocks automated tools like ours from reading it." That's not a GapCite-specific quirk, it's the same gate every AI crawler respects. If GPTBot or ClaudeBot hits the same wall, your page simply never gets read, and a page that's never read can't be cited in an answer, however well it's written.
This usually isn't deliberate. A robots.txt written years ago to block generic scrapers, or copied from a template, or tightened after a bad bot-traffic week, can end up blocking real AI crawlers as a side effect, with nobody noticing until a scan or a server log turns it up.
Check your own robots.txt
Pre-filled with an example that blocks GPTBot specifically, replace it with your own.
The real AI bots this checks
This checks the AI-specific crawlers and on-demand fetchers that actually matter for AI visibility, not general search engines and not SEO tools:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Extended (the AI-specific bot, separate from regular Googlebot)
- Amazon: Amazonbot
- Meta: Meta-ExternalAgent
A bot is reported blocked if its own named User-agent block disallows the root path, or if there's no bot-specific block and a User-agent: * block disallows the root path instead. Real robots.txt parsing has edge cases (wildcards, path-specific rules, multiple Disallow lines); this is a quick, honest self-check against the most common real-world pattern, a root-level block, not a full specification-complete validator.
FAQ
Which AI bots does this check?
GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI), ClaudeBot, Claude-User, and Claude-SearchBot (Anthropic), PerplexityBot and Perplexity-User (Perplexity), Google-Extended (Google's AI training bot, separate from regular Googlebot), Amazonbot, and Meta-ExternalAgent. These are the real AI-specific bots this check looks for, not general search or SEO crawlers.
What if I want to allow AI crawlers?
Remove or narrow any Disallow rule under that bot's own User-agent block, or under User-agent: * if you don't have a bot-specific block. No explicit rule for a bot at all means it's allowed by default.
Does blocking Googlebot affect this?
No. Googlebot is a regular search crawler, not an AI-specific one, so it isn't part of this check. Google-Extended is the separate, AI-specific bot this checker does look for.
Is this the same as blocking SEO crawlers like AhrefsBot?
No. AhrefsBot, SemrushBot, and similar SEO tools are unrelated to this check. This tool only looks for the AI-specific bots listed above.
