Social Crawler

Social Crawler is assigned to the Scraper category by BotScope. Detection is based on the user-agent string. Current pattern:

facebookexternalhit.*Facebot.*Twitterbot|Facebot.*Twitterbot

Generic HTTP clients or scrapers have no official bot status and are mostly used for individual web scraping. They usually don't respect robots.txt and should be blocked as needed.

Once Social Crawler shows up in your logs, BotScope lets you analyze its crawl paths, status codes, IP distribution and temporal activity — by hour, day or week.

This page is being enriched with detail knowledge about Social Crawler.

Similar crawlers in the same category

Discover more crawlers