FR
EN
ES
DE
GPTBot is OpenAI's web crawler that scans the internet to feed ChatGPT's data. Identifiable by the User-Agent "GPTBot/1.0", it can be allowed or blocked via robots.txt.
What is GPTBot?
GPTBot is the web crawler developed by OpenAI. Its role is to scan websites to collect data used to train and power GPT models, particularly ChatGPT. It can be identified in server logs by its User-Agent "GPTBot/1.0".
Should You Allow GPTBot?
The question is strategic for AI visibility:
- Allow GPTBot: Your content feeds ChatGPT, increasing chances of being cited in responses. This is the recommended strategy for GEO
- Block GPTBot: Protects your intellectual property but reduces your visibility on ChatGPT
robots.txt Configuration
Control is done via the robots.txt file:
User-agent: GPTBot
Allow: / # Allow entire site
Disallow: /private/ # Block certain sectionsTracking with AI Labs Audit
AI Labs Audit automatically detects GPTBot visits on your site through the tracking module. You can measure crawl frequency and correlate it with your ChatGPT visibility. See also: ClaudeBot, PerplexityBot, Google-Extended.