Indeed
Indeed is assigned to the Jobs category by BotScope. Detection is based on the user-agent string. Current pattern:
IndeedJobBot
Job aggregator crawlers fetch job postings for platforms like Indeed or StepStone. Crawl frequency grows with the number of your listings.
In the BotScope dashboard you see for Indeed: top paths, status distribution, byte volume and whether the user-agent signature stays stable over time or rotates.
robots.txt directive for Indeed
If you don't want Indeed to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.
User-agent: IndeedJobBot Disallow: /
This page is being enriched with detail knowledge about Indeed.