Add JSON Schema for bot definitions

.github/workflows/ci-validation.yml

@@ -1,6 +1,9 @@
 name: CI validation
 
-on: [push, pull_request]
+on: [push, pull_request, merge_group]
+
+permissions:
+  contents: read
 
 jobs:
   build:
@@ -11,3 +14,8 @@ jobs:
         with:
           node-version: 20
       - run: node validate.js --check
+      - name: Install schema validation dependencies
+        run: npm ci --prefix tools/schema --ignore-scripts
+      - run: node --test schema.test.js tools/schema/validate.test.cjs
+      - name: Validate bot definitions against the JSON Schema
+        run: node tools/schema/validate.cjs

AGENTS.md

@@ -14,13 +14,19 @@ This repository maintains a curated list of well-known bots, crawlers, validator
 
 ## Working with This Repository
 
-### No Package Manager Required
+### Validation Dependencies
 
-This is a **data repository** with NO package.json or dependencies. It uses only Node.js built-in modules.
+This is a **data repository** with no root package.json or project installation.
+The built-in validator and CIDR tests use only Node.js built-in modules. JSON
+Schema validation uses the isolated `tools/schema/` package.
 
-- ❌ **DO NOT** run `npm install`, `npm init`, or any package manager commands
-- ❌ **DO NOT** create a package.json file
-- ✅ **DO** use Node.js directly to run scripts (e.g., `node validate.js --check`)
+- Do not create a root package.json or install dependencies at the repository root.
+- Install validation dependencies with `npm ci --prefix tools/schema --ignore-scripts`.
+- Run schema validation with `node tools/schema/validate.cjs`.
+- Keep validation dependency versions exact and commit the generated lockfile.
+  For an intentional update, run `npm install --prefix tools/schema --ignore-scripts
+  --save-exact <package>@<version>`, review the lockfile, and run
+  `npm audit --prefix tools/schema` plus all validation checks.
 
 ### Validation Script (Critical)
 
@@ -77,8 +83,8 @@ When adding or modifying bot entries, refer to the [README.md](README.md) for:
 
 The repository uses GitHub Actions for validation:
 
-- **Trigger**: Runs on every push and pull request
-- **What it does**: Executes `node validate.js --check`
+- **Trigger**: Runs on every push, pull request, and merge group
+- **What it does**: Installs the locked schema dependencies with scripts disabled, then runs `node validate.js --check`, `node --test schema.test.js tools/schema/validate.test.cjs`, and `node tools/schema/validate.cjs`
 - **Node version**: 20.x
 - **Location**: `.github/workflows/ci-validation.yml`
 
@@ -139,14 +145,13 @@ node validate.js --check
 
 ## Testing Your Changes
 
-Since this is a data repository with no test suite, validation IS the testing:
+Run the built-in validation, regression tests, and JSON Schema validation:
 
 ```bash
-# Run validation - this is your test suite
+npm ci --prefix tools/schema --ignore-scripts
 node validate.js --check
-
-# Expected output: (nothing) with exit code 0
-# If there are errors, they will be printed to stderr
+node --test schema.test.js tools/schema/validate.test.cjs
+node tools/schema/validate.cjs
 ```
 
 ## Tips for Agents

README.md

@@ -40,6 +40,35 @@ To add a new bot to the list, you need to edit the `well-known-bots.json` file a
 5. **Run validation** to ensure your entry is correct: `node validate.js --check`
 6. **Submit a pull request** with your changes
 
+### JSON Schema
+
+The [`well-known-bots.schema.json`](well-known-bots.schema.json) file describes
+the intended structure of `well-known-bots.json` for editors and external JSON
+Schema validators. The repository's built-in `validate.js` script is still the
+source of truth for checks that JSON Schema cannot express, such as compiling
+JavaScript regular expressions and testing `instances` against those patterns.
+
+Install the locked validation dependencies, then validate the JSON file with Ajv:
+
+```bash
+npm ci --prefix tools/schema --ignore-scripts
+node tools/schema/validate.cjs
+```
+
+CI runs both validators and the regression tests:
+
+```bash
+node --test schema.test.js tools/schema/validate.test.cjs
+```
+
+Validation dependencies are isolated in `tools/schema/`. Its lockfile fixes the
+full dependency tree and package integrity hashes; installation disables lifecycle
+scripts. Review dependency and lockfile updates together. To validate a different
+JSON file, pass its path to `node tools/schema/validate.cjs`.
+
+Static IP lists accept IPv4 and IPv6 addresses and CIDR ranges, including IPv6
+ranges with an embedded IPv4 address.
+
 ### Bot Entry Structure
 
 Each entry in the JSON represents a specific bot or crawler and includes the following fields:
@@ -60,6 +89,7 @@ Each entry in the JSON represents a specific bot or crawler and includes the fol
   - **`accepted`** (array): User-Agent strings that should match the pattern
   - **`rejected`** (array): User-Agent strings that should not match
 - **`aliases`** (array): Alternative identifiers for the bot used in other data sources
+- **`description`** (string): Free-form human-readable notes about the bot
 - **`addition_date`** (string): Date the bot was added in YYYY/MM/DD format
 
 ### Available Categories

schema.test.js

@@ -0,0 +1,72 @@
+const assert = require("node:assert/strict");
+const { isIP } = require("node:net");
+const { test } = require("node:test");
+const schema = require("./well-known-bots.schema.json");
+
+const patterns = {
+  4: new RegExp(schema.$defs.ipv4Cidr.pattern, "u"),
+  6: new RegExp(schema.$defs.ipv6Cidr.pattern, "u"),
+};
+
+// Check the schema's CIDR patterns against Node's independent address parser.
+function checkCidr(value, expected) {
+  const parts = value.split("/");
+  const family = isIP(parts[0]);
+  const valid = parts.length === 2 && family !== 0 &&
+    !parts[0].includes("%") && /^(0|[1-9][0-9]*)(?![\s\S])/u.test(parts[1]) &&
+    Number(parts[1]) <= (family === 4 ? 32 : 128);
+  assert.equal(valid, expected, `Node parser: ${JSON.stringify(value)}`);
+  assert.equal(
+    patterns[4].test(value) || patterns[6].test(value),
+    expected,
+    `Schema: ${JSON.stringify(value)}`,
+  );
+}
+
+test("CIDRs accept valid addresses and prefix boundaries", () => {
+  for (const address of ["0.0.0.0", "192.0.2.1", "255.255.255.255"]) {
+    for (let prefix = 0; prefix <= 32; prefix++) {
+      checkCidr(`${address}/${prefix}`, true);
+    }
+  }
+  for (const address of [
+    "::", "::1", "2001:DB8::", "2001:db8:1:2:3:4:5:6",
+    "::ffff:192.0.2.1", "1:2:3:4:5:6:192.0.2.1",
+  ]) {
+    for (let prefix = 0; prefix <= 128; prefix++) {
+      checkCidr(`${address}/${prefix}`, true);
+    }
+  }
+});
+
+test("IPv6 compression accepts every position and rejects too many groups", () => {
+  for (const embedded of [false, true]) {
+    const groups = embedded ? 6 : 8;
+    for (let left = 0; left <= groups + 1; left++) {
+      for (let right = 0; right <= groups + 1; right++) {
+        const before = Array(left).fill("abcd").join(":");
+        const after = [
+          ...Array(right).fill("1234"),
+          ...(embedded ? ["192.0.2.1"] : []),
+        ].join(":");
+        checkCidr(`${before}::${after}/64`, left + right < groups);
+      }
+    }
+  }
+});
+
+test("CIDRs reject malformed addresses and prefixes", () => {
+  for (const value of [
+    "deadbeef/64", "::::/128", "1.2.3.4/64",
+    "192.0.2.1/33", "192.0.2.256/24", "192.00.2.1/24",
+    "2001:db8::/129", "2001:db8::/-1", "2001:db8::/01",
+    "2001:db8::/1.5", "2001:db8::/+64", "2001:db8::/",
+    "2001:db8::/64/64", "1:2:3:4:5:6:7/64", "1:2:3:4:5:6:7:8:9/64",
+    "2001::db8::1/64", "2001:db8:12345::/64", "2001:db8:xyz::/64",
+    "::ffff:256.1.2.3/64", "::ffff:192.00.2.1/64", "fe80::1%eth0/64",
+    " 2001:db8::/64", "2001:db8::/64 ", "2001:db8::/64\n",
+    "192.0.2.1/24\n", "2001:db8::", "192.0.2.1",
+  ]) {
+    checkCidr(value, false);
+  }
+});

tools/schema/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "well-known-bots-schema-validation",
+  "private": true,
+  "dependencies": {
+    "ajv": "8.20.0",
+    "ajv-formats": "3.0.1"
+  }
+}

tools/schema/validate.cjs

@@ -0,0 +1,18 @@
+const { readFileSync } = require("node:fs");
+const { basename, resolve } = require("node:path");
+const Ajv = require("ajv/dist/2020");
+const addFormats = require("ajv-formats");
+
+const schema = require("../../well-known-bots.schema.json");
+const dataPath = process.argv[2] ?? resolve(__dirname, "../../well-known-bots.json");
+const data = JSON.parse(readFileSync(dataPath, "utf8"));
+const ajv = new Ajv({ allErrors: true, strict: true });
+addFormats(ajv);
+const validate = ajv.compile(schema);
+
+if (validate(data)) {
+  console.log(`${basename(dataPath)} valid`);
+} else {
+  console.error(ajv.errorsText(validate.errors, { separator: "\n" }));
+  process.exitCode = 1;
+}

tools/schema/validate.test.cjs

@@ -0,0 +1,56 @@
+const assert = require("node:assert/strict");
+const { spawnSync } = require("node:child_process");
+const { mkdtempSync, rmSync, writeFileSync } = require("node:fs");
+const { tmpdir } = require("node:os");
+const { join } = require("node:path");
+const { test } = require("node:test");
+
+const script = join(__dirname, "validate.cjs");
+
+function validateFixture(t, changes) {
+  const directory = mkdtempSync(join(tmpdir(), "bot-schema-test-"));
+  t.after(() => rmSync(directory, { recursive: true, force: true }));
+  const file = join(directory, "bots.json");
+  const bot = {
+    id: "test-bot",
+    categories: ["monitor"],
+    pattern: { accepted: ["TestBot"], forbidden: [] },
+    verification: [],
+    ...changes,
+  };
+  writeFileSync(file, JSON.stringify([bot]));
+  return spawnSync(process.execPath, [script, file], { encoding: "utf8" });
+}
+
+test("validates the repository data independently of the working directory", () => {
+  const result = spawnSync(process.execPath, [script], {
+    cwd: tmpdir(),
+    encoding: "utf8",
+  });
+  assert.equal(result.status, 0, result.stderr);
+  assert.match(result.stdout, /well-known-bots\.json valid/);
+});
+
+test("rejects structural schema violations", (t) => {
+  const result = validateFixture(t, { categories: ["not-a-category"] });
+  assert.equal(result.status, 1, result.stderr);
+  assert.match(result.stderr, /categories/);
+});
+
+test("enforces the registered IP address formats", (t) => {
+  const result = validateFixture(t, {
+    verification: [{ type: "ip", ips: ["999.999.999.999"] }],
+  });
+  assert.equal(result.status, 1, result.stderr);
+  assert.match(result.stderr, /verification/);
+});
+
+test("rejects the malformed CIDRs from the review through the full schema", (t) => {
+  for (const ip of ["deadbeef/64", "::::/128", "1.2.3.4/64"]) {
+    const result = validateFixture(t, {
+      verification: [{ type: "ip", ips: [ip] }],
+    });
+    assert.equal(result.status, 1, `${ip}: ${result.stderr}`);
+    assert.match(result.stderr, /verification/);
+  }
+});

well-known-bots.json

@@ -2278,7 +2278,6 @@
       "forbidden": []
     },
     "addition_date": "2011/06/21",
-    "url": "",
     "verification": [],
     "instances": {
       "accepted": [
@@ -4273,9 +4272,6 @@
       "forbidden": []
     },
     "url": "http://www.archive.org/details/archive.org_bot",
-    "depends_on": [
-      "heritrix"
-    ],
     "verification": [],
     "instances": {
       "accepted": [
@@ -8933,9 +8929,6 @@
       "forbidden": []
     },
     "addition_date": "2018/10/14",
-    "depends_on": [
-      "libwww-perl"
-    ],
     "verification": [],
     "instances": {
       "accepted": [