# AI crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot and the rest > A reference to the AI crawlers that visit your site (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) and the robots.txt rules for each. By Reece Chown, Agent SEO HQ. Published 11 November 2026. Canonical URL: https://agentseohq.com/articles/ai-crawlers-robots-txt **The short answer:** AI companies run three kinds of crawler: ones that gather **training** data, ones that build a **search** index, and ones that fetch a page because a **user** asked. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended) does not remove you from AI answers. Blocking the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) does. Many sites block all of them without meaning to. ## Three kinds of crawler Before the list, the distinction that makes sense of it. - **Training crawlers** collect pages that may be used to train future models. Whether to allow them is a choice about your content, and reasonable people choose differently. - **Search crawlers** build the index an assistant searches when it answers a question. Block these and you drop out of that assistant's answers. - **User-triggered fetchers** visit a page because a person asked the assistant about it, or pasted in a link. They act more like a browser than a crawler, and some do not follow robots.txt for that reason. ## The reference table The names in the second column are what you write after `User-agent:` in robots.txt. | Company | Robots.txt name | Kind | What blocking it does | |---|---|---|---| | OpenAI | `OAI-SearchBot` | Search | Removes you from ChatGPT search answers | | OpenAI | `ChatGPT-User` | User-triggered | OpenAI says robots.txt "may not apply" to these visits | | OpenAI | `GPTBot` | Training | Keeps your content out of model training | | Anthropic | `Claude-SearchBot` | Search | May reduce your visibility in Claude's search results | | Anthropic | `Claude-User` | User-triggered | Stops Claude fetching your pages when a user asks | | Anthropic | `ClaudeBot` | Training | Keeps future content out of training data | | Perplexity | `PerplexityBot` | Search | Removes you from Perplexity's search results | | Perplexity | `Perplexity-User` | User-triggered | Little: Perplexity says it "generally ignores robots.txt" | | Google | `Googlebot` | Search | Removes you from Google Search, AI Overviews and AI Mode | | Google | `Google-Extended` | Training and grounding control | Opts out of Gemini training and grounding; no effect on Search | | Microsoft | `Bingbot` | Search | Removes you from Bing and Copilot | | Apple | `Applebot` | Search | Removes you from Siri and Spotlight results | | Apple | `Applebot-Extended` | Training control | Opts out of Apple's AI training; search is unaffected | | Meta | `meta-externalagent` | Training | Opts out of Meta's AI crawling | | Common Crawl | `CCBot` | Open archive | Keeps you out of a public dataset widely used for training | | Amazon | `Amazonbot` | Search and AI | Affects Alexa answers | | DuckDuckGo | `DuckAssistBot` | Search | Affects DuckAssist answers | Some details worth knowing: - **Google-Extended and Applebot-Extended are not crawlers.** They are names you can use in robots.txt to control how pages already fetched by Googlebot or Applebot are used. Google states that Google-Extended "does not impact a site's inclusion in Google Search". - **There is no separate crawler for AI Overviews.** They are part of Google Search. The only way to stay out of them is the ordinary Search controls, such as `nosnippet`. - **Old Anthropic names are retired.** Rules for `anthropic-ai` or `Claude-Web` no longer do anything, according to coverage of Anthropic's 2026 documentation update. If your robots.txt has them, it is out of date. - **OpenAI says robots.txt changes take about 24 hours** to apply to search. - **Anthropic honours `Crawl-delay`**, a non-standard line for slowing a crawler down. This list changes several times a year. I checked the OpenAI, Anthropic, Perplexity, Google and Apple rows against those companies' own documentation on 10 November 2026. The other rows reflect what those operators have said about their crawlers and I hold them with less confidence. Check again before relying on any of it. ## Welcome everyone: a robots.txt to copy If you want to be found and recommended, this is the simplest file that says so. It names the crawlers so there is no doubt, keeps private areas private, and points to your sitemap. ```text # AI assistants and search engines are welcome. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /basket/ User-agent: * Allow: / Disallow: /admin/ Disallow: /basket/ Sitemap: https://yourdomain.com/sitemap.xml ``` Strictly, `User-agent: *` with `Allow: /` already lets everyone in. Naming the crawlers is belt and braces: it makes your intent obvious to anyone reading the file, and it stops a later "block everything unknown" rule catching them by accident. Remember that a crawler which finds its own name uses only that group, so repeat your `Disallow` lines there. ## Found in AI search, but not used for training A common and perfectly sensible position. Allow the search and user crawlers, block the training ones: ```text # Search and assistants: yes. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Bingbot User-agent: Googlebot User-agent: Applebot Allow: / # Training: no. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://yourdomain.com/sitemap.xml ``` One trade-off to weigh. A model that never trained on your content knows less about you "from memory", and relies entirely on live search to mention you. For most small businesses that is already the case, so the cost is small. ## The blocks you did not set Robots.txt is only a request, and it is not the only thing between a crawler and your pages. These are the usual culprits when a site blocks AI crawlers by accident. - **Your CDN or security service.** Cloudflare announced in July 2025 that it would block AI crawlers by default for new customers. On two of my own domains I found it adding lines to robots.txt that disallowed GPTBot, ClaudeBot and others, although my own file welcomed them. It took several separate settings to stop it. Other providers have similar features. - **Bot protection.** Features that challenge "non-browser" traffic cannot tell a helpful crawler from a nuisance. Perplexity's documentation specifically warns that Cloudflare's Bot Fight Mode can still challenge its bots whatever your other rules say. - **Security plugins.** Some WordPress plugins ship with lists of "bad bots" that include AI crawlers. - **Hosting defaults.** Some hosts apply their own rules above yours. - **Rate limits.** A crawler that fetches quickly can trip a limit and be shut out for hours. The test is what the crawler actually receives, not what your robots.txt says. Server logs will show it: look for the crawler names above and the status codes next to them. Lots of 403 or 429 against a crawler you meant to allow is your answer. ## Do the crawlers obey? Mostly, with documented exceptions. - OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity all say their automated crawlers follow robots.txt. - User-triggered fetchers are the exception. OpenAI says of ChatGPT-User that robots.txt "may not apply". Perplexity says Perplexity-User "generally ignores" it. Anthropic's general statement that its bots honour robots.txt does not carve out Claude-User. - Not everyone on the internet is as careful. Robots.txt is a convention, not a lock. If content must stay private, put it behind a login. ## A five-minute check 1. Open `yourdomain.com/robots.txt`. Look for any `Disallow: /` and note which `User-agent` it sits under. 2. Look for names you did not add. Lines you do not recognise usually came from a plugin, host or CDN. 3. Check your CDN's bot and AI settings, not just the file. 4. Run the [free checker](https://agentseohq.com/check), which reads your robots.txt and reports on each crawler in the table. 5. If you have access to server logs, search them for `OAI-SearchBot`, `ClaudeBot` and `PerplexityBot` and look at the status codes. **Not sure what is blocking what?** Firewalls and hosting rules do not show up in robots.txt. Send me your address and I will test what the AI crawlers actually receive from your site. It is free. [Get a free Agent SEO check](https://agentseohq.com/contact) or [try the instant checker](https://agentseohq.com/check). ## Frequently asked questions ### Should I block GPTBot? It depends what you want. GPTBot gathers content for training OpenAI's models. Blocking it keeps your content out of training and does not remove you from ChatGPT search, which uses OAI-SearchBot. If your content is your product, blocking is reasonable. If you want to be widely known, allow it. ### Does blocking Google-Extended affect my Google rankings? No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether your content is used for Gemini training and grounding. ### How do I allow ChatGPT to read my website? Make sure robots.txt does not disallow OAI-SearchBot or ChatGPT-User, and that no firewall, security plugin or CDN setting blocks them. OpenAI also recommends allowing requests from its published IP ranges. ### Can I stop my site appearing in Google's AI Overviews? Not with a crawler rule. AI Overviews are part of Google Search and use Googlebot. Google's controls are the standard ones: `nosnippet`, `data-nosnippet`, `max-snippet` or `noindex`, each of which also affects your ordinary search listing. ### Why is my robots.txt blocking AI bots when I never added that? A plugin, your host or your CDN probably did. Cloudflare, for instance, can manage robots.txt and block AI crawlers for you. Compare the file your server holds with the one visitors receive, and check your provider's AI and bot settings. ## Sources - OpenAI, [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots). Checked 10 November 2026. - Anthropic, [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). Checked 10 November 2026. - Perplexity, [Perplexity crawlers](https://docs.perplexity.ai/guides/bots). Checked 10 November 2026. - Google Search Central, [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) and [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features). Checked 10 November 2026. - Apple, [About Applebot](https://support.apple.com/en-us/119829). Checked 10 November 2026. - Search Engine Journal, [Anthropic's Claude bots make robots.txt decisions more granular](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/), 2026. - Search Engine Land, [Cloudflare to block AI crawlers by default](https://searchengineland.com/cloudflare-to-block-ai-crawlers-by-default-with-new-pay-per-crawl-initiative-457708), July 2025. - RFC 9309, [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html).