The User-Agent is an HTTP header sent by a browser or bot with every web request, identifying the software making the request (browser, search engine bot, AI crawler, etc.).
What is a User-Agent?
The User-Agent is a string sent in the HTTP header of every web request. It identifies the software making the request.
AI Bot User-Agents
In the context of AI tracking, User-Agents are essential to identify AI engine crawlers:
GPTBot/1.0— OpenAI (ChatGPT)ClaudeBot/1.0— Anthropic (Claude)PerplexityBot/1.0— Perplexity AIGoogle-Extended— Google (Gemini)
Role in robots.txt
The robots.txt file uses User-Agents to allow or block bot access.