Log Errors Analysis
Every status code in your server logs is a signal. Some are harmless, some cost you visibility, others are security warnings. BotScope filters them by bot, path and time — so you don't dig through noise, you go straight to the cause.
HTTP status codes and what they mean for you
- 200 OK — Fine. Becomes interesting when a specific bot suddenly stops fetching despite 200s — indicates rendering issues.
- 301 / 302 — Redirect. Common crawler pattern: old URLs in sitemap, Googlebot follows 301 chains. More than one hop = crawl budget waste.
- 304 Not Modified — Bot has the resource cached. More 304s is a good sign of efficient caching headers.
- 404 Not Found — Broken links. Bots crawl dead URLs repeatedly. BotScope shows which 404s real search engines hit — fix those or send 410 to permanently retire them.
- 410 Gone — Deliberately removed. More effective than 404 since Google de-indexes the URL.
- 429 Too Many Requests — Your server is throttling a bot. Risky with Googlebot — you lose indexing frequency.
- 500 / 502 / 503 / 504 — Server errors. Bursts of these during bot visits = server under load. Crawlers persistently reduce their rate.
⚠️ 502 Bad Gateway — the underestimated killer
A 502 isn't a "normal" error — it means your reverse proxy (nginx, Cloudflare, load balancer) got no response from the backend (PHP-FPM, Node, Gunicorn). Three typical causes that can be distinguished from the logs:
502s come in bursts during traffic peaks — backend workers exhausted, new requests time out. BotScope shows 502 bursts per hour and which paths are most affected (often API endpoints or compute-heavy pages).
502 on ALL paths simultaneously = backend service crashed or restarted (deploy, OOM kill). BotScope's distribution is then flat across all URLs — no peaks, just a line.
502 only on certain paths (e.g. a new API route or a changed location block) → nginx upstream definition is wrong. Comparing "before/after change" shows it instantly.
🚨 5xx on robots.txt → deindexing risk
Critical: If your server returns /robots.txt with a 5xx status (even briefly), Google may remove your entire domain from the index.
Reason: without a readable robots.txt, Google doesn't know if it's allowed to index the content. When in doubt, Google backs off — and preemptively removes all URLs from the index until robots.txt is available again.
BotScope check: Filter for /robots.txt + status 5xx over the last 30 days. Even a few Googlebot hits are a warning sign — if it lasts an hour, you risk a visibility crash.
Best practice: /robots.txt should be served statically (directly by nginx, not through PHP/Node) and never pass through backend code. That keeps it independent of backend crashes.
📊 Real-world scenario
A shop notices a sudden drop in organic visibility. Search Console shows only generic "Crawling errors". With BotScope you filter:
- Bot: Googlebot (Smartphone)
- Status: 5xx
- Timeframe: last 7 days
Result: 3,200 hits, all on /api/product-stock/ between 03:00 and 04:30. Cause: nightly cron for stock updates locks the DB. Fix: remove the lock — Googlebot resumes normal crawling, rankings recover within 14 days.
What BotScope does for you
- Filters 404s by bot — see only errors real crawlers actually hit
- Burst detection for 5xx clusters — alerts you when a bot starts seeing server errors
- Status distribution per path — instantly spot URLs that fail chronically