Yahoo Slurp
Yahoo Slurp is assigned to the Search category by BotScope. Detection is based on the user-agent string. Current pattern:
Yahoo! Slurp
Search engine crawlers index publicly available content for their results. They typically fetch /robots.txt first and respect crawl-delay and Disallow rules. High crawl frequency usually correlates with good visibility in those search results.
BotScope maps Yahoo Slurp requests onto your session timeline — you spot bursts, can filter them per path, and compare against other bots in the same category.
robots.txt directive for Yahoo Slurp
If you don't want Yahoo Slurp to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.
User-agent: Slurp Disallow: /
This page is being enriched with detail knowledge about Yahoo Slurp.