What OAI-SearchBot is
OpenAI’s description opens with one line: “OAI-SearchBot is for search.” It is “used to surface websites in search results in ChatGPT’s search features”. If your robots.txt opts out of it, your pages “will not be shown in ChatGPT search answers”, although they can still appear as navigational links.
Its user agent starts like an ordinary Chrome browser and ends with compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot, so searching your logs for OAI-SearchBot finds it. After you change robots.txt, OpenAI says it can take about 24 hours for search to reflect it.
OpenAI’s four user agents
OpenAI runs four agents. Its robots.txt controls are the OAI-SearchBot and GPTBot tags, and “each setting is independent of the others”: you can allow OAI-SearchBot to appear in search while disallowing GPTBot.
| User agent | What OpenAI says it is for | robots.txt | IP list |
|---|---|---|---|
OAI-SearchBot |
Showing websites in ChatGPT search results | Follows your rules; changes take about 24 hours | searchbot.json |
GPTBot |
Crawling content that may be used to train OpenAI’s foundation models | Disallowing it says your content should not be used for training | gptbot.json |
ChatGPT-User |
Visiting a page when someone asks ChatGPT or a Custom GPT, and GPT Actions | “robots.txt rules may not apply”; not used to decide what appears in search | chatgpt-user.json |
OAI-AdsBot |
Checking landing pages submitted as ads on ChatGPT | Visits only pages submitted as ads; its data is not used for training | adsbot.json |
So you can stay in ChatGPT search and opt out of training at the same time: allow OAI-SearchBot, disallow GPTBot. Blocking GPTBot does not remove you from search. When OAI-SearchBot or GPTBot fetches robots.txt, OpenAI says it may add a robots.txt marker to the user agent, so you can tell those requests apart in your logs.
For the other AI companies’ crawlers, see AI crawler list: what each bot does and how to block it.
Step 1: allow it in robots.txt
To stay in ChatGPT search but keep your content out of OpenAI’s training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /One rule catches people out. A crawler obeys the group that names it, and falls back to the * group only “if no matching group exists” (RFC 9309). The moment you add a group for OAI-SearchBot, your * rules stop applying to it, so repeat any Disallow lines it should still follow.
Step 2: let its IP addresses through
OpenAI’s help center adds the step most guides miss: “To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI’s published searchbot IP addresses.” The crawler documentation says the same: allow it in robots.txt “and allowing requests from our published IP ranges”.
This matters because robots.txt is only a request: firewalls and CDNs decide who actually gets in, and their rules can turn away a crawler that robots.txt allows. Cloudflare, for example, identifies verified bots by a cryptographic signature or a published IP list, not by the name a request gives. On one of our own sites behind Cloudflare, our openly identified crawler got a 403 while test requests carrying OAI-SearchBot’s name were served: what you see with a faked name tells you little about what the real crawler gets.
What to check:
- On Cloudflare, the AI crawler settings (Search, Training and Agent) and the per-crawler setting in AI Crawl Control. See the Cloudflare section of the AI crawler list.
- On any other firewall or host, rules that block or challenge bots, and any allowlist. If you allowlist, use the ranges in
searchbot.json. The copy we downloaded on 26 October 2026 listed 39 IPv4 ranges; the list can change, so load it automatically instead of pasting it in once.
Step 3: check that it is really OpenAI
A user agent is whatever the client says it is. On citeloop.ai, Cloudflare’s AI Crawl Control showed this for the seven days to 26 October 2026:
| Name in the request | Requests | 2xx | 4xx |
|---|---|---|---|
| OAI-SearchBot | 19 | 14 | 5 |
| GPTBot | 8 | 6 | 1 |
| ChatGPT-User | none listed |
On the Free plan, Cloudflare says AI Crawl Control identifies crawlers by their user agent strings, and the same dashboard showed why that is not proof. A week earlier, every failed request it listed claimed to be Claude-User, and all of them were probing for files such as /.env: scanners borrowing a crawler’s name.
So check the IP address. OpenAI publishes an IP list for each agent; its crawler page does not describe a reverse DNS check. With the short Python script in how to verify Googlebot and AI crawlers reach your site and a downloaded searchbot.json:
python3 check_ip.py searchbot.json 104.210.140.130prints listed, while 104.210.140.144, just outside the listed /28 range, prints NOT listed. We ran both on 26 October 2026. A request that says OAI-SearchBot from an address that is not listed is someone else.
How ChatGPT search finds and ranks pages
OpenAI has said only a little about this, all of it in its help center:
- ChatGPT searches the web on its own when a question would benefit from current information, or when the user chooses Search.
- It “sometimes partners with other search providers”: it rewrites the question into one or more search queries and sends those. The privacy policies OpenAI links for those providers are Microsoft’s and Shopify’s. It does not say which provider handles which query.
- “ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information. Placement is not guaranteed.”
- Answers that use search can include citations, and OpenAI tells users that results and citations “can be incomplete, outdated, or incorrect”.
How do you rank in ChatGPT?
Honestly: nobody outside OpenAI knows. OpenAI has not published its ranking factors, so any list of “ChatGPT ranking factors” is a guess, and no tool or agency can promise a place in the answers.
What you can control is being eligible and easy to use:
- Allow OAI-SearchBot in robots.txt and at your CDN (steps 1 and 2).
- Keep your main content in the HTML. OpenAI does not say whether OAI-SearchBot runs JavaScript, so do not rely on it.
- Answer real questions plainly, with figures and named sources, so a page is worth citing.
Do not try to plant content on other sites to steer AI answers. Google’s spam policies cover “attempting to manipulate generative AI responses”, and Bing’s guidelines list content designed to manipulate the language models used by Bing or Copilot among the practices that can cost a site its visibility or get it removed. See what generative engine optimization is, and what counts as manipulation.
Will you see visitors from ChatGPT?
Look for visits referred by chatgpt.com in your analytics. Expect it to take time: on 26 October 2026, a week after citeloop.ai went live, its Cloudflare Web Analytics showed no visits from chatgpt.com in the past 30 days, even though OAI-SearchBot had already been crawling it.
Citeloop’s free GEO audit reads your robots.txt rules for OAI-SearchBot, ChatGPT-User and five other AI search crawlers on your home page and sampled pages, identifies the CDN in front of your site, and tells you if our openly identified crawler gets blocked. It cannot see your CDN’s allowlists or settings.