AI crawlers: the list, and how to allow or block them

Three kinds of AI crawler

Most AI companies now run separate crawlers for separate jobs, and you can treat each one differently:

• Training crawlers collect pages to train future models. Blocking them keeps your future content out of training data. Examples: GPTBot, ClaudeBot, and the Google-Extended and Applebot-Extended controls.
• Search crawlers build the index that AI search answers draw from. Blocking them is how you disappear from AI search. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot.
• User-triggered fetchers visit a page because a person asked an AI to read it. Several companies say these may not follow robots.txt, because a person, not a crawler, made the request. Examples: ChatGPT-User, Claude-User, Perplexity-User.

The list

“Follows robots.txt” shows what each company’s own page says. “Controlled by robots.txt” means the page tells you to use robots.txt but does not use the word “respects”. IP lists let you check that a request really comes from that company.

Common Crawl’s own pages describe an open dataset of web pages and do not say the data is used for AI training, although many site owners and tools group CCBot with training crawlers.

What blocking each one changes

These are the effects the companies themselves describe:

• OAI-SearchBot: your pages will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI’s settings are independent, so blocking GPTBot does not affect ChatGPT search. Changes take about 24 hours.
• Claude-SearchBot: Anthropic will not index your content for search, which may reduce your visibility in Claude’s search results. Blocking ClaudeBot only keeps future content out of training. Anthropic adds that blocking IP addresses alone may not work correctly.
• PerplexityBot: Perplexity recommends allowing it if you want to appear in its search results.
• Google-Extended: does not affect whether or how you rank in Google Search. It is also not how you leave AI Overviews and AI Mode. Since 31 August 2026, every site can do that in Search Console under Settings, “Search generative AI”. Excluding your site there means no traffic from those features, and it does not change ranking elsewhere or AI training.
• Bing and Copilot: Bing has no separate AI token. Its guidelines say NOARCHIVE keeps content out of Copilot answers and grounding, and NOCACHE limits Copilot to the URL, title and snippet. Bing advises against blocking bingbot in robots.txt.
• Applebot-Extended: keeps your pages out of Apple’s model training only; you can still appear in Apple’s search, and it does not affect ranking.
• DuckAssistBot: opting out does not affect DuckDuckGo’s organic rankings; it takes effect within 72 hours.

robots.txt recipes

Allow everything (an empty Disallow: or no robots.txt at all has the same effect):

User-agent: *
Allow: /

Keep AI search and user fetches, but opt out of training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /

Block just one crawler and leave the rest alone:

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Rules worth knowing when you write these:

• A named group replaces the * group. A crawler uses the group that names it and ignores the * group (RFC 9309; Google says the two are not combined). If you give GPTBot its own group, repeat any rules it should also follow.
• The most specific rule wins, and when an Allow and a Disallow are equally specific, Allow should win.
• User-agent names are case-insensitive.
• Put a robots.txt on every subdomain you want covered; Anthropic’s page says so explicitly.
• Changes are not instant. OpenAI, Perplexity, Meta and Amazon all mention about a day; Amazon may use a robots.txt copy up to 30 days old.

There is no robots.txt rule that blocks only Amazon’s or Bing’s training use: Amazon points to a page-level noarchive, and Bing to NOARCHIVE as above.

If your site is on Cloudflare

Cloudflare changed its AI crawler settings on 15 September 2026, so older guides are out of date:

• The old “Block AI bots” switch and Managed robots.txt were deprecated. Settings are now split into Search, Training and Agent, each with options such as Allow, Block on pages with ads, and Block. Training also has Disallow AI Training.
• Block also stops Googlebot, Bingbot and Applebot, search included. If you only want to opt out of training, use Disallow AI Training, which writes your preference into robots.txt and keeps search crawling.
• New domains, according to that announcement, start with Search, Training and Agent all set to Allow if they do not run ads. Sites with ads start with training disallowed and agents blocked on pages with ads. Some Cloudflare docs pages still describe the earlier defaults, so check what your own dashboard shows.
• AI Crawl Control (formerly AI Audit) is on every plan and shows which AI crawlers visited and whether they were allowed. On the Free plan it identifies crawlers by user agent only.

If you want to be cited, check these settings first: a site can have a perfect robots.txt and still block AI crawlers at the CDN.

What we recommend

For a business that wants to be found and cited by AI search:

1. Allow the search crawlers and user fetchers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and bingbot.
2. Decide on training separately. Opting out of training does not cost you AI search visibility with the companies that separate the two.
3. Check the CDN, not just robots.txt. See how to verify that AI crawlers actually reach your site.

Citeloop’s free GEO audit, opening soon, will read your robots.txt rule by rule for each of these crawlers and check whether a CDN sits in front of your site.