Twitterbot
Twitterbot is assigned to the Social category by BotScope. Detection is based on the user-agent string. Current pattern:
Twitterbot
Social media platforms index content for their internal search and for preview cards in posts. Unlike search engine crawlers they often limit themselves to URLs that were shared on their platform.
BotScope aggregates every Twitterbot request in your session: which URLs it prefers, how many 4xx/5xx responses it gets, whether it crawls from one or many IPs, and what time window it's particularly active in.
robots.txt directive for Twitterbot
If you don't want Twitterbot to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.
User-agent: Twitterbot Disallow: /
This page is being enriched with detail knowledge about Twitterbot.