For an agency, the useful robots.txt question is precise: can the search or retrieval service reach the client's public evidence, while the publisher retains its chosen training policy? These are separate decisions. Allowing a crawler does not guarantee indexing, an AI citation or a recommendation.
Separate search, training and user-requested access
OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in model training. ChatGPT-User fetches pages for some user actions; it is not the automatic search crawler, and OpenAI says robots.txt rules may not apply to those user-initiated requests. Check the current roles and published IP ranges in OpenAI's crawler documentation.
Anthropic likewise separates Claude-SearchBot for search, Claude-User for user-directed retrieval and ClaudeBot for potential training use. Its documentation describes robots.txt controls for these agents. Since 30 October 2026, Anthropic also reports a gradual rollout of a general training-preference signal; explicit ClaudeBot rules take priority. See Anthropic's current crawler guidance. Do not transfer one provider's rules to another.
A policy example, not a universal configuration
The following illustrative groups allow the two named search crawlers and decline the two named training crawlers. They are not a complete robots.txt file and are not a recommendation to change a client's policy without its approval.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
Merge an approved change into the existing rules. Review specific groups as well as the wildcard group, because a matching specific group can change which directives apply. Check every public hostname and subdomain separately. Preserve restrictions for private areas and decide separately whether user-triggered retrieval is permitted. Do not add an Allow rule that exposes an application endpoint intended to require authentication.
Robots.txt controls cooperative crawling, not access security. It is publicly readable and cannot protect confidential files. Use authentication and authorization for private content. Google also distinguishes crawl blocking from preventing a page from appearing in results; a blocked URL may still be known through links. See Google's robots.txt introduction.
An agency verification record that separates the failure layers
Keep a small evidence table for the important pages rather than recording only a global pass or fail:
| Check | Evidence to retain | What it establishes |
|---|---|---|
| Published policy | Hostname, robots.txt body, retrieval date and matching agent group | The instructions served at that time |
| Public HTTP access | Exact URL, redirect chain, final status and response headers | Whether this test request reached a response |
| Readable evidence | Canonical URL, explicit language and relevant text from the HTML | Whether the expected public content was served |
| Observed crawler | Timestamp and source IP verified against the provider's published method | Evidence of a particular provider request |
| Observed answer | Exact prompt, surface, date, response and cited URL | A citation in that observation, not permanent visibility |
A request made with a copied bot User-Agent is a simulation. It can reveal a response difference, but it does not prove that the real provider can reach the page. Conversely, a 403 caused by a WAF or a 429 rate limit is a different finding from a robots directive. Diagnose the layer before changing policy; never remove all protection simply to make a test pass.
Test a change without inventing its impact
Start with one representative public page per language and one deliberately private endpoint. Save the approved policy and the same HTTP evidence before and after the change. Verify that public access works, private access remains protected and the requested language is unchanged by cookies. Then observe real crawler requests and answers separately. A successful fetch is an access result, not proof of indexation or a ranking gain.
AI Labs Audit supports an agency's measurement workflow: our infrastructure, your scenarios, your expertise. Record which models and modes an audit actually used instead of treating a model catalogue as a completed experiment. The methodology and its limits should accompany conclusions shared with clients.
Editorial review: 16 November 2026. Provider documentation was checked on that date. The configuration and verification table above are illustrative working examples, not results from a client deployment.
Questions and answers
How should robots.txt distinguish AI search and training?
Set separate rules for each purpose. OpenAI documents OAI-SearchBot for search and GPTBot for model training. Anthropic distinguishes Claude-SearchBot, ClaudeBot and user-directed Claude-User access. Verify current provider guidance, then test robots.txt and the site’s HTTP/WAF access separately.
Which AI crawlers should I allow?
Choose access by purpose and your content policy. Allowing a search crawler does not guarantee indexing, a citation or a recommendation. Training crawlers are a separate choice; allowing GPTBot is not a prerequisite for ChatGPT search. Check the provider’s current bot documentation rather than allowing every bot as a group.
Does blocking an AI bot always reduce visibility?
No universal effect follows from an unspecified bot name. Search, training and user-directed retrieval serve different purposes. Assess the relevant provider and bot, verify actual access, and measure outcomes. A robots.txt directive is not an authentication or access-control mechanism.
Agencies, consultants and in-house teams: compare the answers generated for your own scenarios, languages and selected models. Our infrastructure, your scenarios, your expertise.