AI answer engines fetch pages with their own crawlers, and many sites block them by accident in robots.txt. Check your file for the crawler names below, allow them on public pages, and turn off any edge setting that rewrites robots.txt for you.
When someone asks ChatGPT, Claude or Perplexity for the best option in your category, the assistant often fetches pages to answer. It does that with its own crawlers, and each one checks your robots.txt first. If the file tells them to stay out, they can't read or cite you, however good your pages are.
The crawlers to look for
These are the AI search and answer crawlers PocketBees checks for:
- OpenAI:
GPTBot,OAI-SearchBot,ChatGPT-User - Anthropic:
ClaudeBot,Claude-User,Claude-SearchBot - Perplexity:
PerplexityBot,Perplexity-User - Training controls:
Google-Extended,Applebot-Extended
Open https://yoursite.com/robots.txt and search for those names. A block like this one keeps that crawler away from every page:
User-agent: GPTBot
Disallow: /
Why it happens by accident
The most common cause we see is not a decision anyone made. On Cloudflare, the Manage your robots.txt setting can rewrite your file at the edge and add rules for AI crawlers. Your own robots.txt looks fine in your code, but what visitors and crawlers receive is different.
Other causes: a template copied from an old project, an SEO plugin default, or a staging rule (Disallow: /) that shipped to production.
What to change
- Allow the AI search and answer crawlers on public pages. Keep private paths such as
/adminor/accountdisallowed. - If you're on Cloudflare, check Security → Settings → Manage your robots.txt and turn it off, or set it to content signals only.
- Fetch the live file again (not the one in your repository) and confirm the block is gone.
Whether you also allow training crawlers such as Google-Extended is a separate choice. Blocking them doesn't stop you being cited in answers.
Try it with your agent
# Task
Edit robots.txt so AI search and answer crawlers (GPTBot,
OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
Claude-SearchBot, PerplexityBot, Perplexity-User) are allowed
on public pages. Keep private paths disallowed.
# Done when
The live robots.txt no longer disallows those crawlers.
PocketBees runs this check on every scan, tells you which crawlers are blocked and where the rule comes from, and re-checks after you fix it.