Free check

Technical

AI crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot and the rest

A reference to the AI crawlers that visit your site (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) and the robots.txt rules for each.

By
· 7 min read

The short answer

AI companies run three kinds of crawler: ones that gather training data, ones that build a search index, and ones that fetch a page because a user asked. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended) does not remove you from AI answers. Blocking the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) does. Many sites block all of them without meaning to.

Three kinds of crawler

Before the list, the distinction that makes sense of it.

  • Training crawlers collect pages that may be used to train future models. Whether to allow them is a choice about your content, and reasonable people choose differently.
  • Search crawlers build the index an assistant searches when it answers a question. Block these and you drop out of that assistant's answers.
  • User-triggered fetchers visit a page because a person asked the assistant about it, or pasted in a link. They act more like a browser than a crawler, and some do not follow robots.txt for that reason.

The reference table

The names in the second column are what you write after User-agent: in robots.txt.

CompanyRobots.txt nameKindWhat blocking it does
OpenAIOAI-SearchBotSearchRemoves you from ChatGPT search answers
OpenAIChatGPT-UserUser-triggeredOpenAI says robots.txt "may not apply" to these visits
OpenAIGPTBotTrainingKeeps your content out of model training
AnthropicClaude-SearchBotSearchMay reduce your visibility in Claude's search results
AnthropicClaude-UserUser-triggeredStops Claude fetching your pages when a user asks
AnthropicClaudeBotTrainingKeeps future content out of training data
PerplexityPerplexityBotSearchRemoves you from Perplexity's search results
PerplexityPerplexity-UserUser-triggeredLittle: Perplexity says it "generally ignores robots.txt"
GoogleGooglebotSearchRemoves you from Google Search, AI Overviews and AI Mode
GoogleGoogle-ExtendedTraining and grounding controlOpts out of Gemini training and grounding; no effect on Search
MicrosoftBingbotSearchRemoves you from Bing and Copilot
AppleApplebotSearchRemoves you from Siri and Spotlight results
AppleApplebot-ExtendedTraining controlOpts out of Apple's AI training; search is unaffected
Metameta-externalagentTrainingOpts out of Meta's AI crawling
Common CrawlCCBotOpen archiveKeeps you out of a public dataset widely used for training
AmazonAmazonbotSearch and AIAffects Alexa answers
DuckDuckGoDuckAssistBotSearchAffects DuckAssist answers

Some details worth knowing:

  • Google-Extended and Applebot-Extended are not crawlers. They are names you can use in robots.txt to control how pages already fetched by Googlebot or Applebot are used. Google states that Google-Extended "does not impact a site's inclusion in Google Search".
  • There is no separate crawler for AI Overviews. They are part of Google Search. The only way to stay out of them is the ordinary Search controls, such as nosnippet.
  • Old Anthropic names are retired. Rules for anthropic-ai or Claude-Web no longer do anything, according to coverage of Anthropic's 2026 documentation update. If your robots.txt has them, it is out of date.
  • OpenAI says robots.txt changes take about 24 hours to apply to search.
  • **Anthropic honours Crawl-delay**, a non-standard line for slowing a crawler down.

This list changes several times a year. I checked the OpenAI, Anthropic, Perplexity, Google and Apple rows against those companies' own documentation on 10 November 2026. The other rows reflect what those operators have said about their crawlers and I hold them with less confidence. Check again before relying on any of it.

Welcome everyone: a robots.txt to copy

If you want to be found and recommended, this is the simplest file that says so. It names the crawlers so there is no doubt, keeps private areas private, and points to your sitemap.

# AI assistants and search engines are welcome.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /basket/

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /basket/

Sitemap: https://yourdomain.com/sitemap.xml

Strictly, User-agent: * with Allow: / already lets everyone in. Naming the crawlers is belt and braces: it makes your intent obvious to anyone reading the file, and it stops a later "block everything unknown" rule catching them by accident. Remember that a crawler which finds its own name uses only that group, so repeat your Disallow lines there.

Found in AI search, but not used for training

A common and perfectly sensible position. Allow the search and user crawlers, block the training ones:

# Search and assistants: yes.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Bingbot
User-agent: Googlebot
User-agent: Applebot
Allow: /

# Training: no.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

One trade-off to weigh. A model that never trained on your content knows less about you "from memory", and relies entirely on live search to mention you. For most small businesses that is already the case, so the cost is small.

The blocks you did not set

Robots.txt is only a request, and it is not the only thing between a crawler and your pages. These are the usual culprits when a site blocks AI crawlers by accident.

  • Your CDN or security service. Cloudflare announced in July 2025 that it would block AI crawlers by default for new customers. On two of my own domains I found it adding lines to robots.txt that disallowed GPTBot, ClaudeBot and others, although my own file welcomed them. It took several separate settings to stop it. Other providers have similar features.
  • Bot protection. Features that challenge "non-browser" traffic cannot tell a helpful crawler from a nuisance. Perplexity's documentation specifically warns that Cloudflare's Bot Fight Mode can still challenge its bots whatever your other rules say.
  • Security plugins. Some WordPress plugins ship with lists of "bad bots" that include AI crawlers.
  • Hosting defaults. Some hosts apply their own rules above yours.
  • Rate limits. A crawler that fetches quickly can trip a limit and be shut out for hours.

The test is what the crawler actually receives, not what your robots.txt says. Server logs will show it: look for the crawler names above and the status codes next to them. Lots of 403 or 429 against a crawler you meant to allow is your answer.

Do the crawlers obey?

Mostly, with documented exceptions.

  • OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity all say their automated crawlers follow robots.txt.
  • User-triggered fetchers are the exception. OpenAI says of ChatGPT-User that robots.txt "may not apply". Perplexity says Perplexity-User "generally ignores" it. Anthropic's general statement that its bots honour robots.txt does not carve out Claude-User.
  • Not everyone on the internet is as careful. Robots.txt is a convention, not a lock. If content must stay private, put it behind a login.

A five-minute check

  1. Open yourdomain.com/robots.txt. Look for any Disallow: / and note which User-agent it sits under.
  2. Look for names you did not add. Lines you do not recognise usually came from a plugin, host or CDN.
  3. Check your CDN's bot and AI settings, not just the file.
  4. Run the free checker, which reads your robots.txt and reports on each crawler in the table.
  5. If you have access to server logs, search them for OAI-SearchBot, ClaudeBot and PerplexityBot and look at the status codes.

Frequently asked questions

Should I block GPTBot?

It depends what you want. GPTBot gathers content for training OpenAI's models. Blocking it keeps your content out of training and does not remove you from ChatGPT search, which uses OAI-SearchBot. If your content is your product, blocking is reasonable. If you want to be widely known, allow it.

Does blocking Google-Extended affect my Google rankings?

No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether your content is used for Gemini training and grounding.

How do I allow ChatGPT to read my website?

Make sure robots.txt does not disallow OAI-SearchBot or ChatGPT-User, and that no firewall, security plugin or CDN setting blocks them. OpenAI also recommends allowing requests from its published IP ranges.

Can I stop my site appearing in Google's AI Overviews?

Not with a crawler rule. AI Overviews are part of Google Search and use Googlebot. Google's controls are the standard ones: nosnippet, data-nosnippet, max-snippet or noindex, each of which also affects your ordinary search listing.

Why is my robots.txt blocking AI bots when I never added that?

A plugin, your host or your CDN probably did. Cloudflare, for instance, can manage robots.txt and block AI crawlers for you. Compare the file your server holds with the one visitors receive, and check your provider's AI and bot settings.

Sources

Keep reading

  • How-to

    How to check if AI can find your website

    A 20-minute check, no tools needed: test whether ChatGPT, Claude, Gemini and Perplexity can reach, read and recommend your website, and what to fix.

    · 7 min read

  • Technical

    What is llms.txt, and does it actually work?

    What llms.txt is, how to write one, and what the evidence says about whether ChatGPT, Claude or Google read it. An honest guide with a working example.

    · 7 min read

  • How-to

    How to get recommended by ChatGPT

    How to get your business recommended by ChatGPT: what OpenAI says it needs, seven things that help, what does not work, and how to check your progress.

    · 7 min read

Find out what the assistants say about you

Send me your address. I check it by hand and reply with what I found, usually within two working days. It costs nothing, and any paid work comes with a fixed quote first.

Get a free Agent SEO checkTry the instant checker