Home › Features › Detect AI Crawlers & Merge Data

Detect AI Crawlers & Merge Data

AI traffic isn't one channel, it's three: crawlers that train models, user-agents that generate answers, and apps that bring real users to your site. BotScope shows all three separately — so you understand what LLMs do with your content and how many visitors come from it.

🤖 The three layers of AI traffic

📚
Training crawlers
GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, PerplexityBot — collect content for model training. BotScope category: AI Crawler.
💬
User-initiated fetches
ChatGPT-User, Claude-User, Perplexity-User, Manus-User, Gemini-Deep-Research — triggered when a user asks a question and the AI fetches your page live. BotScope category: User Bot.
🖱️
Click-through from AI
Real users come from an AI answer to your site. BotScope detects this via referer (chatgpt.com) or UTM parameter (utm_source=chatgpt.com) and marks them as ChatGPT-Click, Perplexity-Click, Claude-Click etc.

🎯 Why this separation matters

Without BotScope all three channels blur in your logs. Each layer has a different meaning for your AI strategy:

Training crawlers
Strategically important — your content lands in the model, influences future answers. But brings no traffic.
User bots
Direct signal: your content is right now in an AI answer. You rank in the AI.
Click-through
The actual visitor. Your real performance KPI.
The correlation reveals the complete picture:
Did a user-bot read my page → was it cited → did the user come?

📊 How BotScope aggregates this

The AI Visibility view stacks per AI platform:

🔍 Bonus: Grok-User detection
Grok (xAI) uses rotating Mac browser fingerprints via residential proxies. BotScope identifies these via burst detection: many IPs from multiple countries within seconds on the same path → flagged as Grok-User.