FR
EN
ES
DE
ClaudeBot is Anthropic's web crawler that scans the internet for Claude's data. Identifiable by the User-Agent "ClaudeBot", it is configurable via robots.txt.
What is ClaudeBot?
ClaudeBot is the web crawler developed by Anthropic to collect data for training and powering the Claude model. Like GPTBot for OpenAI, ClaudeBot scans public websites and respects robots.txt directives.
Identifying ClaudeBot in Your Logs
ClaudeBot presents itself with the User-Agent "ClaudeBot" in HTTP requests. AI Labs Audit automatically detects its visits and integrates them into the AI bot tracking dashboard.
Recommended Configuration
To maximize your visibility on Claude, it is recommended to allow ClaudeBot:
User-agent: ClaudeBot
Allow: /ClaudeBot and Your GEO Strategy
Allowing ClaudeBot is part of a comprehensive GEO strategy. The more your content is accessible to AI crawlers, the more likely you are to be cited in responses. AI Labs Audit measures the correlation between crawl frequency and your Hero Grade on each model.