Drop 1M-line logs → see GPTBot vs OAI-SearchBot cost split → copy the Cloudflare rule. Free, private, $0 vs $139/yr Screaming Frog Log Analyser ($279 Spider) / €171–383/mo JetOctopus (prices verified Feb 2026). No log leaves your machine.
Try Live · 1-click 10k demo · 1GB proof teardown
Enterprise UI (v2.3.0): Material 3 (Material You) theme — top app bar with Analyzer · Knowledge Base ▾ · About · How To Use (guides live on their own pages, not in a sidebar), M3 filled/tonal pill buttons, tonal chips, elevated cards, and linear progress. Centered 1200px layout with no right-side whitespace gap, tables scroll inside their cards, and a crisp SVG brand mark. Sample downloads by URL range with live max-size estimate (0–100 … 1M–10M URLs; in-browser generation caps at 1M, CLI covers the rest), and the 2026 Pixel typeface (Google Sans Flex) throughout.
Private by design: 100% client-side, no XHR/WebSocket/sendBeacon. Only network request is loading the demo file you click. See METHOD.md.
Headless-Chromium run, 2026-09-11, oneGB.jsonl (seed 7). Systematic 1-in-10 sample, honestly bannered.
| Measurement | Result |
|---|---|
| Read + parsed + analyzed | 320,000 of 3,200,000 lines in 6.6 s, 13.0 s wall total |
| Records / bandwidth (sample) | 320,000 records, 10.92 GB, 223,878 unique IPs |
| Traffic split | 70.0% human, 15 categories detected |
| Cost, CloudFront defaults | $2.61 total, $0.25 blockable (training + suspicious only) |
| Response profile | avg 799 ms, P95 1,474 ms, P99 1,535 ms |
| Threats / verification flags | 19,272 suspicious, 427 unusual checks surfaced |
| Crawl-budget waste | 18% trap bandwidth quantified |
Full numbers + 29 screenshots: research/1gb-teardown.html. Fixture: npm run gen-logs -- --seed 42 --days 30 reproduces sample-data/EXPECTED.md byte-identically.
| Tool | Price 2026 | Scale | Privacy | When to use what |
|---|---|---|---|---|
| This tool | $0 | Browser <500MB triage, CLI 500MB–50GB exact, BQ 50GB+ | 100% client-side | 30-sec answer, DPDP/RBI logs that can't leave the machine |
| Screaming Frog LFA | $139/yr (Spider $279/yr) | Desktop, slows past a few M lines | Local desktop | Mature docs, Spider × log joins |
| JetOctopus | €171–383/mo | Cloud millions, logs+crawl+GSC joined | Cloud upload | Continuous monitoring, history, client dashboards |
| OnCrawl / Botify | €49–2,500/mo | Up to 500M lines/day | Cloud + FTP | Enterprise SSO/procurement |
| Splunk | Custom ingest | Infinite | Cloud | Build-it-yourself DIY |
Honest take: paid tools win on continuity/history. This wins on privacy + speed + price. Details: Frog vs JetOctopus vs $0.
Bot DB v2026.09.17, 127 signatures. Interviewers ask exactly this.
| Class | Examples | Edge action |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Bytespider | Block/throttle freely, 60/min (20 aggressive) |
| Search-index | OAI-SearchBot, PerplexityBot, Claude-SearchBot | ALLOW 120/min, 429 + Retry-After only — never hard block |
| User-triggered | ChatGPT-User, Perplexity-User, OAI-AdsBot (allow-listed) | Do NOT throttle (300/min abuse ceiling only) |
Includes Cloudflare Sept-15-2026 defaults, Pay Per Crawl 402 beta, Web Bot Auth, vendor IP JSON (data/bot-ips.json, 14 sources, full CIDRs + IPv6). See BOTS.md.
| # | Module | What you get |
|---|---|---|
| 1 | Bot Classification | 127 sigs + 18 browser patterns, bot-first order, bandwidth/IPs/TTFB per category |
| 2 | Verification (4 layers) | Vendor IP JSON → TLS → cloud-ASN heuristic → behavior; UNVERIFIED spoof KPI, opt-in DoH rDNS |
| 3 | Crawl Budget + Render Gap | 16 traps (gclid, cart, /_next/data/) + JS-shell <5KB hunt (zero GEO citation chance) |
| 4 | Cost | Measured bytes × CDN preset, 95% interval, blockable = training + suspicious only |
| 6 | Edge Rules + Policy | Cloudflare/Fastly/AWS copy-paste, robots + llms.txt + crawlers.json bundle, OAI-AdsBot guard |
| 7–8 | Performance + Traffic | TTFB bots vs humans, bursts (z-score, no deps), 404-cluster tickets, GSC overlay |
| 9 | Security | Traversal, .git/HEAD, creds, SSRF, velocity anomalies |
| 10 | CFO / FinOps | Totals, monthly/annual run-rate, CSV + redesigned 2-page PDF (M3 blue, numbered sections, KPI cards, dynamic exec box, aligned tables + sign-off) — vendored jsPDF, no CDN. Full Report PDF ↓ exports all 10 modules + CFO summary in one offline reference file |
Screenshots: assets/screenshots/09-tab1-classification.png → 29-cfo-viewport.png (analysis modules unchanged; v2.2.1 refreshed the app shell centering, brand mark, and CFO PDF design). Full history: CHANGELOG.md.
git clone https://github.com/dipakjad1993/Log-File-Bot-Traffic-Cost-Analyzer.git
cd Log-File-Bot-Traffic-Cost-Analyzer
node dev-server.js # LOCAL DEV ONLY — http://localhost:8080. Never deploy to Pages.
# or: docker build -t log-analyzer . && docker run -p 8080:8080 log-analyzer
npm test # 121 asserts
npm run lintInputs: JSON array, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended (#Fields), AWS ALB, Cloudflare Logpush, .gz — mixed files + rotations OK. Streams in 8MB slices; 500MB+ exact → node tools/cli.js --exact --in access.log --out summary.json. Test data: pick a URL range in the tool (0–100 / 100–1K / 1K–10K / 10K–100K / 100K–1M / 1M–10M, each showing its max file size, ≈450 bytes/row; browser builds up to 1M) or npm run gen-logs -- --lines N --out file.jsonl.
Deploy: Cloudflare Pages (primary, main, no build command, headers from _headers) or GitHub Pages backup. Live: https://log-file-bot-traffic-cost-analyzer.pages.dev
- Cost = measured bytes × pricing. Blockable = training + suspicious only (asserted in tests). Origin-compute OFF by default.
- Verification = vendor IP JSON first, heuristics labeled low-confidence, never verdicts. Sampling disclosed with stride/counts/timing.
- Limits: Browser <500MB triage (300k sampled) · CLI 500MB–50GB exact · BigQuery 50GB+. See METHOD.md.
flowchart LR
Drop[Drop logs .gz/multi-file] --> Slice[8MB slices + auto-detect]
Slice --> Worker[Web Worker analyze 14 steps]
Worker --> Render[10 renderers: bots/verify/cost/edge/CFO]
Render --> Export[CFO PDF + CSV + Cloudflare rules + llms.txt]
Worker --> CLI[CLI --exact 500MB-50GB]
CLI --> BQ[BigQuery pack 50GB+]
BOTS.md · METHOD.md · ROADMAP.md · BIGQUERY.md · sitemap.xml
CONTRIBUTING.md · SECURITY.md · MIT (LICENSE) — PRs welcome (good first issue labeled). If this saved you €383/mo, please star the repo.