OnCrawl

OnCrawl is assigned to the SEO category by BotScope. Detection is based on the user-agent string. Current pattern:

www\.oncrawl\.com

SEO tool crawlers build internal databases for tools like Ahrefs, Semrush, Sistrix or Majestic. They have no direct SEO impact — only the tool accesses the data. Crawl frequency grows with your domain authority.

BotScope maps OnCrawl requests onto your session timeline — you spot bursts, can filter them per path, and compare against other bots in the same category.

robots.txt directive for OnCrawl

If you don't want OnCrawl to crawl your site, add the following block to your /robots.txt. This works only for bots that honour robots.txt — malicious crawlers ignore it.

User-agent: OnCrawl
Disallow: /

This page is being enriched with detail knowledge about OnCrawl.

Popular crawlers by category

SearchGooglebot-Image Googlebot-Video Googlebot-News

AI CrawlersGoogle-Extended Google-NotebookLM Google-Lens

User BotsChatGPT-User Claude-User Perplexity-User

Link PreviewGoogleImageProxy Chrome Privacy Preserving Prefetch Proxy Google Web Preview

AppsSlack-App Claude-App Grok-App

Social MediaFacebook Twitterbot LinkedInBot

MonitoringBotScope Google-Firebase UptimeRobot

SecurityGoogle-Safety Google-Site-Verification VisionHeight

AdvertisingAdsBot-Google Mediapartners-Google Google-Display-Ads-Bot

Price Comparisonbilliger.de idealo Geizhals

PaymentStripe Mollie PayPal

LegalComplianceChecker Cookiebot CookieYes

JobsLinkupBot Indeed Jooble

OfficeGoogle Docs Microsoft Office LibreOffice Apache OpenOffice

PlagiarismTurnitin Copyleaks Copyscape

ScrapersGoogle-Read-Aloud Feedfetcher Google-Apps-Script

Plugin/CMSJetpack Cleantalk WordPress.com

Similar crawlers in the same category

Schema-Markup-ValidatorSEO Google-Structured-DataSEO AhrefsSEO SemrushSEO SiteAuditBotSEO SplitSignalBotSEO

Discover more crawlers

imgproxyScraper Social CrawlerScraper OAI-SearchBotAI Chefkoch-AppApps MistralAI-UserUser-Bot DynatraceMonitor