AI Crawlers and robots.txt: What Brands Should Allow

If you want your brand to show up in AI answers, allow the search crawlers: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot (Google Search). Whether you also allow the training crawlers, GPTBot, ClaudeBot and the Google-Extended token, is a separate choice: as of 2026-10-16 each company documents its training and search agents separately, and OpenAI and Google say outright that opting out of training is not the same as opting out of their search.

How we checked: on 2026-10-16 we read the crawler pages published by OpenAI, Anthropic, Perplexity and Google, plus Google's robots.txt documentation. Every quote below comes from those pages.

Training bots, search bots and user fetchers

The four companies describe three different kinds of visitor:

• Training crawlers collect content that may be used to train AI models: GPTBot, ClaudeBot, and Google-Extended (a control token, not a separate crawler). Perplexity's page says neither of its agents is used to collect content for training AI foundation models.
• Search crawlers index pages so the assistant can find and link them when answering: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot for Google Search.
• User-triggered fetchers visit a page because a person asked the assistant something: ChatGPT-User, Claude-User and Perplexity-User. Their robots.txt treatment differs by company, as the sections below show.

For a brand, the search crawlers are the ones that decide whether your pages can be found and linked in AI search answers. Allowing them does not mean you will be cited; it only means you are not excluding yourself.

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

OpenAI's crawler overview lists the user agents and says each setting is independent. On search:

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

On training:

“Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.”

The same page gives the example of allowing OAI-SearchBot to appear in search while disallowing GPTBot. It notes that if both are allowed, OpenAI may use one crawl for both uses, and that for search it “can take ~24 hours from a site's robots.txt update for our systems to adjust.”

ChatGPT-User is different. OpenAI says it is “not used for crawling the web in an automatic fashion” and that “because these actions are initiated by a user, robots.txt rules may not apply.” It also says ChatGPT-User is not used to decide whether content appears in search; OAI-SearchBot is the token for that. The page also lists OAI-AdsBot, which only visits pages submitted as ads on ChatGPT.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic's help article on crawling describes three bots and what happens when you disable each one. ClaudeBot collects content that could contribute to model training:

“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.”

Note the word “future”. Claude-SearchBot indexes content for search, and Anthropic says disabling it “may reduce your site's visibility and accuracy in user search results.” Claude-User fetches pages when a person asks Claude a question; Anthropic says disabling it prevents retrieval in response to a user query and “may reduce your site's visibility for user-directed web search.”

Anthropic states that its bots honor “industry standard directives in robots.txt”, support the non-standard Crawl-delay extension, and need the robots.txt rule repeated on every subdomain you want to opt out. It warns against relying on IP blocking instead, because that stops its bots from reading your robots.txt file.

Perplexity: PerplexityBot and Perplexity-User

Perplexity's crawler page describes PerplexityBot as a search crawler:

“PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.”

Perplexity recommends allowing PerplexityBot in robots.txt if you want to appear in its search results. Perplexity-User handles fetches that people trigger, and here the page is direct: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity says changes may take up to 24 hours to be reflected.

Google: Googlebot and Google-Extended

Google's list of common crawlers says these crawlers “always obey robots.txt rules when crawling automatically.” Rules addressed to Googlebot affect Google Search. Google-Extended works differently: it has no HTTP user-agent string of its own, and the token is used only in robots.txt to control whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google is explicit about Search:

“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Rules for Googlebot affect Google Search, so blocking Googlebot is not a way to opt out of Gemini training without also affecting Search. Google-Extended is the token Google documents for the Gemini side.

A robots.txt for brands that want to be found

A robots.txt for a brand that wants to be findable in AI search while opting out of model training could look like this. If you are happy for your public pages to be used in training, simply leave out the last three groups.

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Two details from Google's documentation are worth knowing before you edit:

• Google's robots.txt specification says “only one group is valid for a particular crawler”: the most specific matching user agent wins and the * group is not combined with it. If you name a bot, repeat inside its group any Disallow lines from your * group that should still apply to it.
• The same spec says rules apply only to the host, protocol and port where the file is served, so blog.example.com needs its own file.

Google's robots.txt introduction also warns that “different crawlers interpret syntax differently” and that robots.txt cannot enforce behavior: “it's up to the crawler to obey them.” It is a crawl signal, not access control.

How to check what your robots.txt allows

1. Open https://yourdomain.com/robots.txt in a browser, then repeat for every subdomain that serves content you care about.
2. Look for a User-agent: * group with Disallow: /. Unless a bot has its own group, that line tells it to stay out of the whole site.
3. Search the file for each token above. A named group overrides * for that bot, so check that it says what you intend.
4. In Google Search Console, open the robots.txt report. It shows the robots.txt files Google found for the top 20 hosts on your site, when they were last crawled, and any parsing errors or warnings. It works only for domain-level properties.
5. Check your firewall or CDN bot settings. Perplexity notes that sites behind a web application firewall may need to allow its bots explicitly. OpenAI, Anthropic and Perplexity publish IP lists you can use to verify that a request claiming to be their bot really is.
6. After a change, give it time: OpenAI mentions about 24 hours for search and Perplexity up to 24 hours.

What this does not cover: whether any AI company may use your content under copyright or other law, and how content collected before you changed robots.txt is treated beyond what the pages above say. This is not legal advice; check each company's page again before you rely on it, because these lists change.

Agent Agency is software that learns your product from a link and runs AI agents that promote it in online communities.