Add documented bots and refresh verification methods

[davidmytton]

Summary

Corpus 635 → 793. Bots with verification 79 → 85.

Bots added (158) from getarcis/well-known-bots, monperrus/crawler-user-agents, official docs, and sources cited by arXiv 2603.28546v1. Entries require a loadable source and a distinctive, substantiated User-Agent token.

Patterns widened so live UAs match: Checkly, Perplexity User, AmazonAdBot, Qwant, New Relic Synthetics, Datadog Synthetic, Montastic.

Verification review (every existing method, plus GoodBots as a map of first-party feeds — third-party GitHub raw lists were not added):

• Re-fetched all 41 unique remote URLs: all HTTP 200; selectors still extract addresses.
• Added first-party CIDR/IP/DNS:
  • Applebot
  • DuckDuckBot
  • DuckAssistBot
  • Common Crawl / CCBot
  • UptimeRobot IPv4 / IPv6
  • Telegram CIDR
  • MojeekBot
  • Googlebot-Mobile
  • Better Uptime / Better Stack
• Meta geofeed URLs: http:// → HTTPS.
• censys-inspect: added current ranges from Censys docs.
• Removed unpublished Adagio IPs and discontinued Freshping IP verification.

Still valid as-is:
Google,
Bing,
Ahrefs,
OpenAI,
Claude,
Perplexity,
Stripe,
Datadog,
New Relic,
Pingdom,
Kagi,
Quantcast,
Oncrawl,
Algolia,
Kimi.
Yandex stays DNS-only because it does not publish IPs.

Not added as verification:
Semrush (no consecutive blocks),
Twitter/X (no first-party feed),
WebPageTest (HTML table only),
Marginalia (outdated),
GoodBots raw lists.

Review follow-up

• Removed unsupported DeepSeekBot, ThousandEyes, Playwright, Puppeteer, and Selenium entries.
• Renamed AutoRAG to cloudflare-ai-search; matches current Cloudflare-AI-Search and legacy Cloudflare-AutoRAG. Source
• Corrected Amzn-User/0.1 to Amazon’s published UA. Amazon’s IP page is HTML with embedded JSON, so it is not added as an unsupported http-json source. Source
• Replaced the bare Cypress sample with a full Cypress Electron UA and linked Cypress browser docs.
• New Google/WatchFor selectors use the README-correct JSONPath form.
• Tightened Anomura/ and Poduptime;\s*\+fediverse\.observer patterns.

Test plan

• node validate.js --check (exit 0)
• git diff --check (exit 0)
• Every accepted instance matches all accepted patterns and no forbidden patterns
• All 41 verification URLs HTTP 200; new machine-readable feeds extract prefixes/IPs
• Updated Amazon, Cloudflare, and Cypress documentation URLs return HTTP 200

  

[arcjet-review]

Arcjet Review — 🟢 Low Risk

Decision: Checked

Rationale: Data-only change to a bot registry JSON file. No code, no auth, no secrets, no dependencies. Description is thorough and matches the diff. Size is large but homogeneous (bot record additions plus small pattern/URL edits). No escalation triggers fire. One selector-escaping inconsistency is worth the author's attention but does not warrant blocking — the file remains schema-valid and other selectors in the PR are correct.

Summary of Changes

Adds 163 new bot definitions and extends verification (CIDR/IP/DNS) on ~10 existing bots (Googlebot, DuckDuckGo, Apple, CommonCrawl, Mojeek, Telegram, UptimeRobot, Better Uptime, Freshping, censys-inspect). Widens UA regexes for Checkly, Perplexity-User, AmazonAdBot, Qwant, NewRelic Synthetics, Datadog Synthetic, Montastic. Switches Facebook geofeed URLs from http to https. Removes unpublished Adagio static IPs.

Notes

Diff is ~4k lines, above the 500-line size threshold, but the content is homogeneous JSON records so automated review remains reliable. Several widened regex patterns (Checkly, AmazonAdBot with no trailing delimiter, [Pp]uppeteer/[Ss]elenium/[Cc]ypress/[Pp]laywright) will now match any UA containing those substrings — likely intentional given this is a bot registry, but worth the maintainer's attention if false positives matter downstream.

The AI assessed this PR as approvable, but the trust level (1) does not allow auto-approval. A human reviewer must approve this PR.

_{Review: 9f36478c | Model: anthropic/claude-opus-4-7 | Powered by Arcjet Review}