AI crawlers: which bots read your site, and how to check they can

An AI crawler is a bot that fetches your pages so an AI system can read them — either to answer a user's question right now, or to build the index it answers from later. If your site blocks these bots, no amount of content or structured data matters: the model simply never sees you. This is the single most common, and most fixable, cause of AI invisibility we find in audits.

The crawlers that matter

User agent	Who runs it	What it does
GPTBot	OpenAI	Collects pages for model training
OAI-SearchBot	OpenAI	Builds the index behind ChatGPT search results
ChatGPT-User	OpenAI	Fetches a page live when a user's question needs it
ClaudeBot	Anthropic	Crawls for Claude
PerplexityBot	Perplexity	Indexes pages for Perplexity answers
Google-Extended	Google	Controls use of your content in Gemini and AI features
Bingbot	Microsoft	Powers Bing, Copilot — and parts of ChatGPT search
Amazonbot, meta-externalagent, cohere-ai	Amazon, Meta, Cohere	Assistant and model crawlers

Two distinctions worth understanding. First, training crawlers vs answer crawlers: GPTBot collects data for future models, while ChatGPT-User and OAI-SearchBot affect whether you can be cited today. Blocking the first is a business decision about your content; blocking the second is simply opting out of AI search. Second, Google-Extended is not a crawler at all — it's a permission token. Google's normal Googlebot still crawls you; the token only says whether that content may feed Gemini and AI features. Details are in Google's crawler documentation and OpenAI's bot documentation.

How to check in 60 seconds

Open yoursite.com/robots.txt in a browser. You are looking for two things:

1. Explicit blocks. Any line pairing one of the agents above with Disallow: / means that system is locked out entirely.
2. A blanket block. A User-agent: * group with Disallow: / blocks everything that isn't explicitly allowed elsewhere — including every AI crawler.

Also check for a noindex meta tag on key pages (view source, search for "noindex") and, if you can, an X-Robots-Tag header. A page can be perfectly crawlable and still be excluded by either.

The three ways sites block AI by accident

Security plugins. WordPress security and anti-scraping plugins often ship "block AI bots" as a default-on feature. It's presented as protection; in practice it's an opt-out of AI search.

CDN bot protection. Cloudflare and similar services have one-click toggles for blocking AI crawlers. Someone enables it for good reasons — bandwidth, content scraping — and nobody connects it to the marketing goal of being recommended by ChatGPT.

Copy-pasted robots.txt. Templates from a few years ago block whole ranges of user agents. They get inherited from site to site, long after anyone remembers why.

What we recommend allowing

For almost every business whose goal is customers, the answer is: allow the answer-layer crawlers without hesitation (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Bingbot, Google-Extended). Being present in answers is exactly what you're trying to buy with all your other marketing.

Training crawlers (GPTBot and friends) are a genuine judgement call for publishers whose content is the product. For a service business, blocking them buys nothing and costs future recognition — models that learned your name during training can mention you even when browsing is off.

A minimal, permissive robots.txt looks like this:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

After you change it

Crawlers re-read robots.txt on their own schedule — typically within days. To speed things up on the Bing side (which also feeds ChatGPT search), submit your URLs via IndexNow. Then verify in your server or CDN logs that the bots actually return: if GPTBot and PerplexityBot start appearing in your access logs, the door is genuinely open.

FAQ

Does allowing AI crawlers hurt my SEO?

No. They are separate systems from Googlebot's ranking crawl. Allowing them adds AI visibility without touching your search rankings.

Will AI bots overload my server?

For a normal business site, their traffic is negligible — a few requests per day. High-volume publishers may want a crawl-delay; a ten-page service site will never notice.

How do I know it worked?

Run the 5-minute visibility test a couple of weeks later and compare. Crawler access is step 2 of that test for a reason: nothing downstream works without it.