Anthropic AI
Anthropic AI is assigned to the AI category by BotScope. Detection is based on the user-agent string. Current pattern:
(Claude-Web|anthropic-ai)
AI training crawlers collect web content as training material for large language models. They bring no direct traffic — your content ends up in the model and influences future AI answers. Most honour robots.txt, some are more aggressive than search engine crawlers.
BotScope maps Anthropic AI requests onto your session timeline — you spot bursts, can filter them per path, and compare against other bots in the same category.
robots.txt directive for Anthropic AI
If you don't want Anthropic AI to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.
User-agent: anthropic-ai Disallow: /
This page is being enriched with detail knowledge about Anthropic AI.