Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- **Local Web Bot Auth verification** (RFC 9421 HTTP Message Signatures, tag `web-bot-auth`).
- `detectBot(request)` — verify an inbound request's agent signature and get a `verified` / `impersonation` / `claimed` / `none` verdict (with agent name/category for verified agents). Accepts a WHATWG `Request` or `{ method, url, headers }`.
- `webBotAuth()` rule — denies impersonation of known agents in the rules engine by default; `onImpersonation` / `onClaimed` / `allowCategories` options.
- Cached, curated directory client (Ed25519 + RSA-PSS-SHA512, JWK-thumbprint keyids); zero network on the warm path, no SSRF surface. Runs on Node and Vercel Edge / WinterCG runtimes.
- New exports: `AgentVerifier`, `createAgentVerifier`, `DirectoryCache`, `DEFAULT_SIGNED_AGENT_DIRECTORIES`, and types `AgentVerdict`, `AgentStatus`, `AgentCategory`, `WebBotAuthConfig`, `AgentVerifierOptions`, `SignedAgentDirectory`.
- Doc: "Verify AI agents with Web Bot Auth in Next.js" (`docs/verify-ai-agents-web-bot-auth.md`).

## [0.4.0] - 2026-06-30

### Added
Expand Down
32 changes: 32 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,36 @@ const wd = new WebDecoy({

- **`rateLimit({ max, window, algorithm?, action?, keyBy? })`** — fixed or sliding window, keyed by IP (or a custom function). No key.
- **`tripwire({ paths?, prefixes?, patterns?, includeDefaults? })`** — deterministic honeypot-path detection. No key.
- **`webBotAuth({ onImpersonation?, onClaimed?, allowCategories? })`** — verify AI-agent signatures (Web Bot Auth / RFC 9421) locally; deny impersonators of known agents. No key.
- **`filter({ expression, action? })`** — an expression language over IP reputation/geo (e.g. `ip.tor`, `ip.country in ["CN", "RU"]`). Requires an API key for enrichment.

## Verify AI agents (Web Bot Auth)

AI agents like OpenAI's Operator now **cryptographically sign** their requests
([Web Bot Auth](https://datatracker.ietf.org/wg/webbotauth/about/), RFC 9421).
WebDecoy verifies those signatures **locally** — on Node and on the edge, with no
API key and no network on the warm path — so you can tell a real verified agent
from someone forging its identity.

```typescript
import { WebDecoy } from '@webdecoy/node';

const wd = new WebDecoy();

const verdict = await wd.detectBot(request); // a WHATWG Request, or { method, url, headers }
// verdict.status: 'verified' | 'impersonation' | 'claimed' | 'none'
if (verdict.status === 'impersonation') return new Response('Forbidden', { status: 403 });
if (verdict.status === 'verified') console.log('verified agent:', verdict.agentName, verdict.category);
```

Or drop it into the rules engine to deny impersonation automatically:

```typescript
const wd = new WebDecoy({ rules: [webBotAuth()] }); // denies known-agent impersonation
```

Full guide: [**Verify AI agents with Web Bot Auth in Next.js**](docs/verify-ai-agents-web-bot-auth.md).

## What counts as a false positive?

The zero-false-positive claim is about *design*, and it holds if you know the edge cases:
Expand Down Expand Up @@ -200,6 +228,10 @@ Deterministic honeypot-path detection. `tripwire()` returns a `Rule` for the `ru

Additional local rules for the `rules` array. `filter()` requires an API key for IP enrichment.

### `webBotAuth(config?)` / `detectBot(request)`

Local Web Bot Auth verification (RFC 9421). `webBotAuth()` returns a `Rule` that denies agent impersonation; `detectBot(request)` returns the verdict directly for custom handling. See the [Web Bot Auth guide](docs/verify-ai-agents-web-bot-auth.md). Exported types: `AgentVerdict`, `AgentStatus`, `AgentCategory`, `WebBotAuthConfig`, `AgentVerifierOptions`, `SignedAgentDirectory`.

### `protect(metadata, options?): Promise<ProtectResult>`

Full analysis of a request (platform feature). Returns a decision:
Expand Down
171 changes: 171 additions & 0 deletions docs/verify-ai-agents-web-bot-auth.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
# Verify AI agents with Web Bot Auth in Next.js

AI agents that browse the web on a user's behalf — OpenAI's Operator, ChatGPT's
browsing tool, and a growing cohort behind the IETF
[`webbotauth`](https://datatracker.ietf.org/wg/webbotauth/about/) working group —
now **cryptographically sign** their requests using
[Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/)
(RFC 9421 HTTP Message Signatures, tag `web-bot-auth`). A signed request proves
"this really is that agent," and an *unverifiable* signature that claims a known
agent's identity is a forgery you can block.

`@webdecoy/node` verifies these signatures **locally, in your middleware** — on
Node and on every WinterCG edge runtime (Vercel Edge, Cloudflare Workers). No
API key, and **no network on the warm path**: trusted agent directories are
fetched once and cached, so a verification is a header parse, a map lookup, and
one WebCrypto check (well under 5 ms).

> This is the same verification WebDecoy runs at ingest and at the edge — the
> SDK, the edge validator, and the backend all share the
> [`github.com/WebDecoy/web-bot-auth`](https://github.com/WebDecoy/web-bot-auth)
> profile and speak the same verdict taxonomy.

## The verdict

Every request resolves to one of four statuses:

| `status` | Meaning | Typical action |
|---|---|---|
| `verified` | Signature validated against a trusted agent's published key. `agentName` and `category` are populated. | Allow (optionally with elevated trust) |
| `impersonation` | A signature claimed a **known** agent's key but failed verification (bad signature, or outside its validity window). A forgery. | **Deny** |
| `claimed` | A signature is present but unverifiable — an unknown/uncurated signer, or malformed. Not proof of an agent, nor of a forgery. | Let your other rules decide |
| `none` | No Web Bot Auth signature. Ordinary traffic. | Continue |

## Option A — `detectBot()` in Next.js middleware

`detectBot()` is the low-level primitive. It takes a WHATWG `Request` (what
Next.js middleware and Edge routes already hand you) and returns the verdict, so
you decide what to do:

```typescript
// middleware.ts
import { NextRequest, NextResponse } from 'next/server';
import { WebDecoy } from '@webdecoy/node';

const wd = new WebDecoy(); // no API key needed for local verification

export async function middleware(req: NextRequest) {
const verdict = await wd.detectBot(req);

switch (verdict.status) {
case 'impersonation':
// Someone is forging a known agent's identity — block it.
return new NextResponse('Forbidden', { status: 403 });

case 'verified':
// A real, cryptographically verified agent. Let it through and tag it
// for your route handlers.
const res = NextResponse.next();
res.headers.set('x-verified-agent', verdict.agentName ?? '');
res.headers.set('x-verified-agent-category', verdict.category ?? '');
return res;

default:
return NextResponse.next();
}
}

export const config = { matcher: ['/((?!_next/static|favicon.ico).*)'] };
```

`detectBot()` also accepts a plain `{ method, url, headers }` object for
non-`Request` environments (Node HTTP, Express):

```typescript
const verdict = await wd.detectBot({
method: req.method,
url: `https://${req.headers.host}${req.url}`,
headers: req.headers as Record<string, string>,
});
```

## Option B — the `webBotAuth()` rule

If you already use WebDecoy's rules engine, add `webBotAuth()` to your `rules`
array. It runs alongside your rate limits and tripwires and, by default, **denies
impersonation** — no branching code required. Verification happens automatically
before the rule evaluates.

```typescript
// middleware.ts
import { withWebDecoy } from '@webdecoy/nextjs';
import { webBotAuth, rateLimit } from '@webdecoy/node';

export default withWebDecoy({
rules: [
webBotAuth(), // deny agent impersonation
rateLimit({ max: 100, window: 60 }),
],
});

export const config = { matcher: ['/api/:path*'] };
```

An impersonation attempt is denied with a `403` before it reaches your app. The
verdict is also surfaced on `ProtectResult.agent` for `withBotProtection`/manual
`protect()` callers.

### Rule options

```typescript
webBotAuth({
onImpersonation: 'DENY', // default; forged known-agent signature
onClaimed: 'ALLOW', // default; unverifiable/unknown signer
allowCategories: ['ai_crawlers'], // optional: only these verified categories pass
dryRun: false, // record the verdict but never block
});
```

- Set `onClaimed: 'DENY'` to require that **every** signed request come from an
agent you trust (an unknown signer is then treated as a violation).
- `allowCategories` lets you accept, say, verified search engines but not
AI crawlers — a verified agent outside the list is handled per `onClaimed`.

## Trusted directories (and why there's no SSRF)

Verification is **curated**: the SDK only ever fetches the well-known directories
of agents on an allowlist — never a URL taken from the incoming request's
`Signature-Agent` header. That means a hostile request can't make your
middleware fetch an arbitrary origin (no SSRF), and the warm path stays on
in-memory keys.

The default list tracks the agents that sign production traffic today (OpenAI
Operator, ChatGPT). Override it — for example to add your own signed crawlers —
via the constructor:

```typescript
const wd = new WebDecoy({
webBotAuth: {
directories: [
{ name: 'OpenAI', category: 'ai_crawlers', directory: 'https://operator.openai.com' },
{ name: 'Acme Crawler', category: 'monitoring', directory: 'https://crawler.acme.example' },
],
cacheTtlMs: 6 * 60 * 60 * 1000, // stale-while-revalidate; default 6h
},
});
```

A directory publishes its keys at
`/.well-known/http-message-signatures-directory` as a JWK Set (Ed25519 / OKP and
RSA supported). Keys are matched by their RFC 7638/8037 JWK thumbprint — the
value agents put in the signature's `keyid` — never by a mutable `kid` label.

## Runtime support

- **Vercel Edge Middleware / Cloudflare Workers** — verified against a real
Vercel Edge Runtime VM in the SDK's test suite. Uses only `crypto.subtle`,
`fetch`, `Request`/`Headers`, `URL`, and `atob` — no Node built-ins.
- **Node ≥ 18** — global `fetch` and WebCrypto Ed25519.

## How it relates to the platform

Local verification is free and needs no key. With a WebDecoy API key, verified
and impersonation verdicts also flow into your dashboard alongside TLS
fingerprinting, IP reputation, and detection analytics — so you can see *which*
agents visit and *who* tried to impersonate them over time.

## See also

- [`github.com/WebDecoy/web-bot-auth`](https://github.com/WebDecoy/web-bot-auth) — the underlying Go/TS profile
- [RFC 9421 — HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421)
- [IETF `webbotauth` working group](https://datatracker.ietf.org/wg/webbotauth/about/)
Loading