Scrapy

Scrapy is assigned to the Scraper category by BotScope. Detection is based on the user-agent string. Current pattern:

Scrapy

Generic HTTP clients or scrapers have no official bot status and are mostly used for individual web scraping. They usually don't respect robots.txt and should be blocked as needed.

BotScope maps Scrapy requests onto your session timeline — you spot bursts, can filter them per path, and compare against other bots in the same category.

robots.txt directive for Scrapy

If you don't want Scrapy to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.

User-agent: Scrapy
Disallow: /

This page is being enriched with detail knowledge about Scrapy.

Similar crawlers in the same category

Discover more crawlers