robots.txt for AI: Search, Training and Access Controls

For an agency, the useful robots.txt question is precise: can the search or retrieval service reach the client's public evidence, while the publisher retains its chosen training policy? These are separate decisions. Allowing a crawler does not guarantee indexing, an AI citation or a recommendation.

Separate search, training and user-requested access

OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in model training. ChatGPT-User fetches pages for some user actions; it is not the automatic search crawler, and OpenAI says robots.txt rules may not apply to those user-initiated requests. Check the current roles and published IP ranges in OpenAI's crawler documentation.

Anthropic likewise separates Claude-SearchBot for search, Claude-User for user-directed retrieval and ClaudeBot for potential training use. Its documentation describes robots.txt controls for these agents. Since 30 October 2026, Anthropic also reports a gradual rollout of a general training-preference signal; explicit ClaudeBot rules take priority. See Anthropic's current crawler guidance. Do not transfer one provider's rules to another.

A policy example, not a universal configuration

The following illustrative groups allow the two named search crawlers and decline the two named training crawlers. They are not a complete robots.txt file and are not a recommendation to change a client's policy without its approval.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

Merge an approved change into the existing rules. Review specific groups as well as the wildcard group, because a matching specific group can change which directives apply. Check every public hostname and subdomain separately. Preserve restrictions for private areas and decide separately whether user-triggered retrieval is permitted. Do not add an Allow rule that exposes an application endpoint intended to require authentication.

Robots.txt controls cooperative crawling, not access security. It is publicly readable and cannot protect confidential files. Use authentication and authorization for private content. Google also distinguishes crawl blocking from preventing a page from appearing in results; a blocked URL may still be known through links. See Google's robots.txt introduction.

An agency verification record that separates the failure layers

Keep a small evidence table for the important pages rather than recording only a global pass or fail:

CheckEvidence to retainWhat it establishes
Published policyHostname, robots.txt body, retrieval date and matching agent groupThe instructions served at that time
Public HTTP accessExact URL, redirect chain, final status and response headersWhether this test request reached a response
Readable evidenceCanonical URL, explicit language and relevant text from the HTMLWhether the expected public content was served
Observed crawlerTimestamp and source IP verified against the provider's published methodEvidence of a particular provider request
Observed answerExact prompt, surface, date, response and cited URLA citation in that observation, not permanent visibility

A request made with a copied bot User-Agent is a simulation. It can reveal a response difference, but it does not prove that the real provider can reach the page. Conversely, a 403 caused by a WAF or a 429 rate limit is a different finding from a robots directive. Diagnose the layer before changing policy; never remove all protection simply to make a test pass.

Test a change without inventing its impact

Start with one representative public page per language and one deliberately private endpoint. Save the approved policy and the same HTTP evidence before and after the change. Verify that public access works, private access remains protected and the requested language is unchanged by cookies. Then observe real crawler requests and answers separately. A successful fetch is an access result, not proof of indexation or a ranking gain.

AI Labs Audit supports an agency's measurement workflow: our infrastructure, your scenarios, your expertise. Record which models and modes an audit actually used instead of treating a model catalogue as a completed experiment. The methodology and its limits should accompany conclusions shared with clients.

Editorial review: 16 November 2026. Provider documentation was checked on that date. The configuration and verification table above are illustrative working examples, not results from a client deployment.

Questions and answers

How should robots.txt distinguish AI search and training?

Set separate rules for each purpose. OpenAI documents OAI-SearchBot for search and GPTBot for model training. Anthropic distinguishes Claude-SearchBot, ClaudeBot and user-directed Claude-User access. Verify current provider guidance, then test robots.txt and the site’s HTTP/WAF access separately.

Which AI crawlers should I allow?

Choose access by purpose and your content policy. Allowing a search crawler does not guarantee indexing, a citation or a recommendation. Training crawlers are a separate choice; allowing GPTBot is not a prerequisite for ChatGPT search. Check the provider’s current bot documentation rather than allowing every bot as a group.

Does blocking an AI bot always reduce visibility?

No universal effect follows from an unspecified bot name. Search, training and user-directed retrieval serve different purposes. Assess the relevant provider and bot, verify actual access, and measure outcomes. A robots.txt directive is not an authentication or access-control mechanism.

About the author

Davy Abderrahman

Founder of

As a specialist in visibility in AI answers (AEO/GEO/LLMO), I help agencies and consultants measure and analyze their clients’ presence on ChatGPT, Claude, Gemini, Perplexity and other AI answer engines.

AEO GEO LLMO AI visibility AI audits
How do your brands appear in AI answers?

Agencies, consultants and in-house teams: compare the answers generated for your own scenarios, languages and selected models. Our infrastructure, your scenarios, your expertise.

Was this article helpful?

- (0 votes)