Guide to OpenAI bots/user-agents: understanding the ChatGPT ecosystem

For most internet users, ChatGPT looks like a single interface able to answer any question. In reality, behind every answer lies a technical infrastructure made of several bots, each handling a specific task on the web. Understanding OpenAI bots user-agent is therefore not a detail reserved for developers: it has become a full-fledged visibility lever in its own right for any search strategy in the age of artificial intelligence.

OpenAI does not deploy a single crawler, but three distinct user agents that step in at different moments of the information journey. Knowing which to allow, which to block and how they read your content directly shapes your presence in ChatGPT’s answers and, more broadly, in generative search engines.

Good news: this mechanism is nothing like a black box. Here is how it works, in concrete terms.

Key takeaways from this article

• OpenAI does not deploy a single bot, but three distinct user agents: GPTBot, OAI-SearchBot and ChatGPT-User.
• Each bot has its own mission: training the models, indexing for search, or visiting on a user’s request.
• The robots.txt file lets you control each user agent independently of the others.
• Blocking GPTBot protects your training data without harming your visibility in ChatGPT’s answers.
• The other artificial-intelligence bots (Claude, Perplexity, Google) follow a comparable logic.

What is a user-agent and why do OpenAI bots matter for SEO?

A user agent is the identity card a piece of software presents to every web server it visits. When a browser or a bot loads a page, it sends a string announcing its name, its version and sometimes its purpose. Engines like Google, Bing or OpenAI’s models rely on this signal to identify themselves, and site owners use it to decide who may access their content.

Definition: user agent

A user agent is a string sent by a client (browser, application or crawler) during an HTTP request. It lets the server identify the nature of the visitor. For artificial-intelligence bots, this same identifier serves as a control key in the robots.txt file.

Reading an OpenAI user-agent string: the compatible, KHTML, like Gecko format

OpenAI’s bots present themselves with a standardized string, readable in your server logs. GPTBot, for example, announces itself like this: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot. There you find the historical mention compatible agent KHTML, like Gecko, inherited from browsers, followed by the bot’s real name and a link to its documentation. The Gecko compatible portion does not mean it is a disguised Firefox: it is a compatibility convention that most crawlers keep. The information useful for SEO sits at the end of the string, in the GPTBot/1.1 token, which precisely identifies the bot and its version.

The three OpenAI bots you need to know

GPTBot: the model-training bot

GPTBot is OpenAI’s best-known crawler. Its mission is large-scale data collection for model training, from GPT-4 to the generations that follow. It crawls the web like a classic crawler, extracts text, phrasing and factual information, then feeds the datasets that form the system’s base intelligence. Its user agent is GPTBot/1.1. If you refuse to let your content serve for training, you can block it via your robots.txt file. Key point: this block does not erase your presence in ChatGPT’s answers, because indexing for search depends on another bot.

OAI-SearchBot: the indexer for conversational search

OAI-SearchBot is the bot dedicated to web search within ChatGPT. Unlike GPTBot, it does not collect data for training: it discovers and indexes pages so they can be cited as sources in conversational search results. Its oai user agent is OAI-SearchBot/1.0. If your goal is to appear as a reference when a user queries the ChatGPT SearchBot, this bot absolutely must be allowed. Blocking it means disappearing from the citations displayed in the answers, even if your pages stay visible elsewhere.

ChatGPT-User: the agent triggered by the user

ChatGPT-User resembles neither GPTBot nor the SearchBot. It is not an automatic crawler: it triggers occasionally, when a user, or a custom GPT via Actions, asks ChatGPT to consult a specific URL. It then behaves more like a browser than an indexing bot, in order to retrieve fresh information: an opening time, a price, an updated figure. Its ChatGPT user agent is ChatGPT-User/1.0, with more recent versions observed in some logs. Like most human-triggered agents, it does not apply robots.txt directives the same way an automatic crawler does, since the visit stems from a deliberate action.

Summary table of OpenAI user agents

How the three agents work together: a typical journey

To grasp how these bots complement each other, let’s follow a query from end to end. Picture a user searching for the up-to-date pricing of a SaaS tool.

Upstream, GPTBot has already come across this sector during its crawl campaigns: the model knows the nature of the product and the existence of the vendor, thanks to the data collection carried out months earlier for model training. This general knowledge lets ChatGPT immediately understand the context of the question.

If the user activates web search, OAI-SearchBot comes into play: having indexed the pricing page, it lets ChatGPT offer it as a relevant source in its answer. The user gets a contextualized answer, paired with a reliable link.

Finally, if the question concerns a precise and potentially volatile price, the model may judge the indexed information too old. It then triggers ChatGPT-User, which visits the page live to extract the most recent figure. Three bots, three time frames, one coherent answer.

What has changed recently in the OpenAI ecosystem

The adjustments made in late 2025 clarified the role of each bot and their implications for publishers.

• Refocusing of OAI-SearchBot: its official purpose is now search and indexing for answers, not model training.
• Special status of ChatGPT-User: acting as a direct user agent, it no longer complies with robots.txt directives the way a systematic crawler would.
• Shared crawl: when GPTBot and OAI-SearchBot are both allowed, OpenAI can reuse a single pass for both uses, which reduces the load on your servers.
• Extension to GPTs: ChatGPT-User now handles requests coming from custom GPTs and Actions, a volume set to grow.
• Multiple versions: several version numbers of the same user agent can coexist in your logs, a sign of a fast-evolving ecosystem.

Configuring robots.txt for OpenAI bots

The robots.txt file placed at the root of your site remains the first control lever. Each user agent is steered independently there, which allows fine-grained strategies according to your priorities.

Maximizing your visibility in ChatGPT

To appear as much as possible in the ecosystem, allow OAI-SearchBot and ChatGPT-User. The first guarantees the indexing of your pages for conversational answers, the second enables real-time checks. This is the recommended configuration for editorial sites and brands that want to be cited in ChatGPT’s answers.

Refusing training while staying visible

If you do not wish to feed entraînement des modèles without sacrificing your visibility, block GPTBot and let OAI-SearchBot through. This balanced approach keeps you in ChatGPT’s résultats de recherche without surrendering your content to the training dataset. Allow about twenty-four hours before a change to robots.txt is reflected in search behavior.

Checking that your bots really get through

Allowing a bot in robots.txt is not always enough. Many sites unintentionally block crawlers via their application firewall or request limits that return errors. Remember too to validate the authenticity of visitors: a user agent can be spoofed, and only a check of the IP address ranges published by OpenAI confirms that a request truly comes from its servers. The regular analysis of your logs, a pillar of rigorous on-site technical optimization, remains the best way to know which bots actually crawl your site.

OpenAI versus the other AI bots: Claude, Perplexity, Google

The OpenAI ecosystem is not isolated. The other players in artificial intelligence have adopted a very similar logic, separating training, indexing and on-demand visiting. On Anthropic’s side, ClaudeBot collects the training data, while Claude SearchBot indexes for retrieval and Claude-User answers user-triggered visits. Perplexity, for its part, distinguishes PerplexityBot, its indexing bot, from Perplexity-User for human queries. Google, finally, separates Googlebot from Google-Extended, the latter controlling the use of your content for training its generative models. The lesson is clear: thinking in terms of bot families, rather than by brand, is the right lens for steering your visibility across all LLMs.

Doko, your partner for ranking in AI engines

Doko is a human-scale Lyon-based webmarketing agency, based in La Mulatière. For more than ten years, we have helped companies generate qualified traffic and revenue through their website. As a Google Partner Premier, we work on SEO, SEA, Google Ads, Meta Ads and analytics.

Generative search does not remove the underlying logic of SEO: it adds a technical layer that few companies master. Correctly configuring your user agents, structuring your content for the bots and measuring your citations in ChatGPT are among the work areas we build into our methodologies. We do not promise miracles: we work on real data and we adjust continuously.

Want to know how to steer OpenAI’s bots and turn this visibility into concrete results? Let’s talk.

FAQ: OpenAI bots and user-agents

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content for model training, whereas OAI-SearchBot indexes your pages so they are cited in ChatGPT’s search. The two are controlled separately in the robots.txt file.

Does blocking GPTBot make my site disappear from ChatGPT?

No. Blocking GPTBot only prevents the use of your content for training. Your visibility in the answers depends on OAI-SearchBot, which stays active as long as you allow it.

Does ChatGPT-User respect the robots.txt file?

ChatGPT–User acts on a user’s request and behaves like a browser. It does not apply robots.txt directives the same way an automatic crawler does, since the visit results from a deliberate human action.

How can you check that a bot is really a genuine OpenAI bot?

A user agent can be imitated. To confirm the origin of a request, compare the IP address of the visitor with the ranges published by OpenAI in its official files. This is the only reliable method against spoofing.

Do other AIs like Claude or Perplexity use the same bots?

No, but the logic is identical. Claude, Perplexity et Google each deploy their own user agents, split between training, indexing and on-demand visiting. A coherent strategy takes them all into account.