Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,11 @@ const wd = new WebDecoy({ rules: [webBotAuth()] }); // denies known-agent impers

Full guide: [**Verify AI agents with Web Bot Auth in Next.js**](docs/verify-ai-agents-web-bot-auth.md).

[Run the signed-agent proof](playgrounds/signed-agent/README.md): generated Ed25519
signatures, a tampered target, an expired signature, an unknown signer and an
unsigned control. Uses the published SDK with a controlled public-key directory;
no provider traffic or detection-accuracy claim.

## Clearance enforcement on Vercel

With a WebDecoy account, the paths you protect can require a clearance token, checked in middleware before the page renders. A real browser earns one automatically; a scraper that never runs JavaScript never gets the page. It works with any framework on Vercel, follows the site's WebDecoy mode (monitor until you switch to enforce) and fails open.
Expand Down
233 changes: 118 additions & 115 deletions docs/verify-ai-agents-web-bot-auth.md
Original file line number Diff line number Diff line change
@@ -1,174 +1,177 @@
# Verify AI agents with Web Bot Auth in Next.js
# Verify signed AI-agent requests in Next.js and Node.js

A familiar User-Agent name is a claim. For supported signed requests,
`@webdecoy/node` can check a cryptographic signature against keys from configured
trusted agent directories and return a verdict you can inspect in your app.
Local verification needs no WebDecoy API key.

This guide covers the SDK's supported `web-bot-auth` signature profile, built on
[RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421).
Signing support and request coverage vary by agent. An unsigned request is not
proof of a human, and a valid signature is not permission to access a route.

## Run the proof first

The [standalone signed-agent example](../playgrounds/signed-agent/README.md)
generates a demo Ed25519 key, signs real request components and runs the
published SDK verifier. It prints these controlled results:

| Request | SDK status | Reason |
| --- | --- | --- |
| Valid signature from the demo trusted key | `verified` | — |
| Covered target URL changed after signing | `impersonation` | `bad-signature` |
| Trusted-key signature outside its validity window | `impersonation` | `expired` |
| Signature from an unknown key | `claimed` | `unknown-key` |
| No signature | `none` | — |

```bash
cd playgrounds/signed-agent
npm ci --workspaces=false
npm run dev
npm test
```

AI agents that browse the web on a user's behalf — OpenAI's Operator, ChatGPT's
browsing tool, and a growing cohort behind the IETF
[`webbotauth`](https://datatracker.ietf.org/wg/webbotauth/about/) working group —
now **cryptographically sign** their requests using
[Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/)
(RFC 9421 HTTP Message Signatures, tag `web-bot-auth`). A signed request proves
"this really is that agent," and an *unverifiable* signature that claims a known
agent's identity is a forgery you can block.
Requires Node.js 20+. No monorepo build or API key is needed. Only the directory
response is supplied by an injected fixture containing the generated public
key. Signing and verification are real; SDK verdicts are not mocked. These are
not captured requests from OpenAI, Google or another provider, and this is not
a complete protocol-conformance or production-accuracy test.

`@webdecoy/node` verifies these signatures **locally, in your middleware** — on
Node and on every WinterCG edge runtime (Vercel Edge, Cloudflare Workers). No
API key, and **no network on the warm path**: trusted agent directories are
fetched once and cached, so a verification is a header parse, a map lookup, and
one WebCrypto check (well under 5 ms).
## Understand the verdict

> This is the same verification WebDecoy runs at ingest and at the edge — the
> SDK, the edge validator, and the backend all share the
> [`github.com/WebDecoy/web-bot-auth`](https://github.com/WebDecoy/web-bot-auth)
> profile and speak the same verdict taxonomy.
- **`verified`:** the signature validates against a key in the configured trusted
directories. The result includes the agent name, category and key ID.
- **`impersonation`:** a known-key signature did not pass verification or its
validity checks. Inspect `reason` before deciding how to respond.
- **`claimed`:** the signature cannot be verified with the available keys or
supported format. An unknown signer is not proof of abuse.
- **`none`:** no applicable signature was recognized. Other detection and policy
checks still apply.

## The verdict
`impersonation` is the SDK's classification. An expired signature or a modified
covered component can also reflect clock or proxy issues. It does not, by
itself, establish the requester's intent.

Every request resolves to one of four statuses:
## Observe in Next.js middleware

| `status` | Meaning | Typical action |
|---|---|---|
| `verified` | Signature validated against a trusted agent's published key. `agentName` and `category` are populated. | Allow (optionally with elevated trust) |
| `impersonation` | A signature claimed a **known** agent's key but failed verification (bad signature, or outside its validity window). A forgery. | **Deny** |
| `claimed` | A signature is present but unverifiable — an unknown/uncurated signer, or malformed. Not proof of an agent, nor of a forgery. | Let your other rules decide |
| `none` | No Web Bot Auth signature. Ordinary traffic. | Continue |
Install the core SDK:

## Option A — `detectBot()` in Next.js middleware
```bash
npm install @webdecoy/node
```

`detectBot()` is the low-level primitive. It takes a WHATWG `Request` (what
Next.js middleware and Edge routes already hand you) and returns the verdict, so
you decide what to do:
Start by recording the outcome without changing route access:

```typescript
// middleware.ts
import { NextRequest, NextResponse } from 'next/server';
import { WebDecoy } from '@webdecoy/node';

const wd = new WebDecoy({}); // no API key needed for local verification
const wd = new WebDecoy({});

export async function middleware(req: NextRequest) {
const verdict = await wd.detectBot(req);

switch (verdict.status) {
case 'impersonation':
// Someone is forging a known agent's identity — block it.
return new NextResponse('Forbidden', { status: 403 });

case 'verified':
// A real, cryptographically verified agent. Let it through and tag it
// for your route handlers.
const res = NextResponse.next();
res.headers.set('x-verified-agent', verdict.agentName ?? '');
res.headers.set('x-verified-agent-category', verdict.category ?? '');
return res;

default:
return NextResponse.next();
}
console.info('agent verification', {
status: verdict.status,
reason: verdict.reason,
agent: verdict.agentName,
});
return NextResponse.next();
}

export const config = { matcher: ['/((?!_next/static|favicon.ico).*)'] };
export const config = { matcher: ['/api/:path*'] };
```

`detectBot()` also accepts a plain `{ method, url, headers }` object for
non-`Request` environments (Node HTTP, Express):
`detectBot()` returns evidence; it does not block a request. Keep authorization,
rate limits and your existing edge protection in place. After observing legitimate
traffic, decide which outcomes your application should deny, challenge or allow.

```typescript
const verdict = await wd.detectBot({
method: req.method,
url: `https://${req.headers.host}${req.url}`,
headers: req.headers as Record<string, string>,
});
```
For custom Node.js handlers, the same API accepts
`{ method, url, headers }`. Use the original absolute request URL and covered
header values. Reconstructing the wrong public scheme, host, port or path behind
a proxy can invalidate a signature; accept forwarding information only from
proxies you actually trust.

## Use the verdict in the rules engine

## Option B — the `webBotAuth()` rule
If you already use the Next.js adapter:

If you already use WebDecoy's rules engine, add `webBotAuth()` to your `rules`
array. It runs alongside your rate limits and tripwires and, by default, **denies
impersonation** — no branching code required. Verification happens automatically
before the rule evaluates.
```bash
npm install @webdecoy/nextjs @webdecoy/node
```

```typescript
// middleware.ts
import { withWebDecoy } from '@webdecoy/nextjs';
import { webBotAuth, rateLimit } from '@webdecoy/node';

export default withWebDecoy({
rules: [
webBotAuth(), // deny agent impersonation
rateLimit({ max: 100, window: 60 }),
],
rules: [webBotAuth(), rateLimit({ max: 100, window: 60 })],
});

export const config = { matcher: ['/api/:path*'] };
```

An impersonation attempt is denied with a `403` before it reaches your app. The
verdict is also surfaced on `ProtectResult.agent` for `withBotProtection`/manual
`protect()` callers.

### Rule options
The middleware starts in **monitor mode**. `webBotAuth()` returns a denial for
known-key verification failures by default, but a denial verdict does not itself
mean an HTTP403 response in monitor mode. Inspect `req.webdecoyDecision` in
supported handlers. Switch to `mode: 'enforce'` only after reviewing your traffic.
The in-memory rate limit is per process; use a shared store across replicas.

```typescript
webBotAuth({
onImpersonation: 'DENY', // default; forged known-agent signature
onClaimed: 'ALLOW', // default; unverifiable/unknown signer
allowCategories: ['ai_crawlers'], // optional: only these verified categories pass
dryRun: false, // record the verdict but never block
onImpersonation: 'DENY',
onClaimed: 'ALLOW',
// Optional: decide which verified categories this rule accepts.
allowCategories: ['ai_crawlers'],
});
```

- Set `onClaimed: 'DENY'` to require that **every** signed request come from an
agent you trust (an unknown signer is then treated as a violation).
- `allowCategories` lets you accept, say, verified search engines but not
AI crawlers — a verified agent outside the list is handled per `onClaimed`.

## Trusted directories (and why there's no SSRF)
A verified agent outside `allowCategories` is handled according to `onClaimed`.
With the default `onClaimed: 'ALLOW'`, that list is not a blocking policy. Set
policy options deliberately rather than treating category labels as permissions.

Verification is **curated**: the SDK only ever fetches the well-known directories
of agents on an allowlist — never a URL taken from the incoming request's
`Signature-Agent` header. That means a hostile request can't make your
middleware fetch an arbitrary origin (no SSRF), and the warm path stays on
in-memory keys.
## Trust and directory fetching

The default list, `DEFAULT_SIGNED_AGENT_DIRECTORIES`, tracks the agents that
sign production traffic today (ChatGPT, Google Agent, WebDecoyBot) and matches
the list the WebDecoy edge validator trusts. Passing `directories` replaces it,
so spread the defaults in to add your own signed crawlers:
The SDK fetches keys from configured directory URLs. It does not choose a fetch
target from the incoming request's `Signature-Agent` header. Keep that trusted
list under application control.

```typescript
import { WebDecoy, DEFAULT_SIGNED_AGENT_DIRECTORIES } from '@webdecoy/node';

const wd = new WebDecoy({
webBotAuth: {
directories: [
...DEFAULT_SIGNED_AGENT_DIRECTORIES,
{ name: 'Acme Crawler', category: 'monitoring', directory: 'https://crawler.acme.example' },
],
cacheTtlMs: 6 * 60 * 60 * 1000, // stale-while-revalidate; default 6h
directories: [...DEFAULT_SIGNED_AGENT_DIRECTORIES],
fetchTimeoutMs: 5000,
},
});
```

A directory publishes its keys at
`/.well-known/http-message-signatures-directory` as a JWK Set (Ed25519 / OKP and
RSA supported). Keys are matched by their RFC 7638/8037 JWK thumbprint — the
value agents put in the signature's `keyid` — never by a mutable `kid` label.
Passing `directories` replaces the defaults. A cold lookup can await directory
fetching; cached keys are reused and stale caches may refresh in the background.
Do not promise zero network traffic for every verification. Directory reachability
and key availability affect which requests can be verified.

## Runtime support
No latency or detection-accuracy claim is established by the local example.
The Node.js fixture uses Node's crypto module to generate signatures; it does
not establish browser or edge-runtime compatibility for every provider's profile.

- **Vercel Edge Middleware / Cloudflare Workers** — verified against a real
Vercel Edge Runtime VM in the SDK's test suite. Uses only `crypto.subtle`,
`fetch`, `Request`/`Headers`, `URL`, and `atob` — no Node built-ins.
- **Node ≥ 18** — global `fetch` and WebCrypto Ed25519.
## Keep identity and permission separate

## How it relates to the platform
WebDecoy complements Cloudflare and other edge protections. A signature check
at the app can inspect only requests reaching that middleware. Verification does
not give an agent permission to scrape, purchase, create an account or bypass
your rate limits. It also does not establish that a person is present or that
an AI system cited your site.

Local verification is free and needs no key. With a WebDecoy API key, verified
and impersonation verdicts also flow into your dashboard alongside TLS
fingerprinting, IP reputation, and detection analytics — so you can see *which*
agents visit and *who* tried to impersonate them over time.
Local outcomes are separate from hosted dashboard reporting. Configuring an API
key enables platform features; keyless `detectBot()` output is not evidence of
a cloud detection.

## See also
## References and next steps

- [`github.com/WebDecoy/web-bot-auth`](https://github.com/WebDecoy/web-bot-auth) — the underlying Go/TS profile
- [RFC 9421 — HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421)
- [IETF `webbotauth` working group](https://datatracker.ietf.org/wg/webbotauth/about/)
- [HTTP Message Signatures, RFC 9421](https://www.rfc-editor.org/rfc/rfc9421)
- [IETF Web Bot Auth working group](https://datatracker.ietf.org/wg/webbotauth/about/)
- [Cloudflare's Web Bot Auth integration profile](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/)
- [SDK verifier implementation](../packages/webdecoy/src/agent/verifier.ts)
- [Interactive Express tripwire playground](../playgrounds/express/README.md)
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@
"build": "turbo run build",
"dev": "turbo run dev",
"lint": "turbo run lint",
"test": "turbo run test && npm run test:playground",
"test": "turbo run test && npm run test:playground && npm run test:signed-agent",
"clean": "turbo run clean && rm -rf node_modules",
"format": "prettier --write \"**/*.{ts,tsx,js,jsx,json,md}\"",
"check:edge": "npm run check:edge -w @webdecoy/node -w @webdecoy/nextjs -w @webdecoy/hono",
"test:playground": "npm ci --prefix playgrounds/express --ignore-scripts --no-audit --no-fund && npm test --prefix playgrounds/express"
"test:playground": "npm ci --prefix playgrounds/express --ignore-scripts --no-audit --no-fund && npm test --prefix playgrounds/express",
"test:signed-agent": "npm ci --prefix playgrounds/signed-agent --ignore-scripts --no-audit --no-fund && npm test --prefix playgrounds/signed-agent"
},
"devDependencies": {
"@changesets/cli": "^2.27.1",
Expand Down
45 changes: 45 additions & 0 deletions playgrounds/signed-agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Verify an AI-agent signature locally

A reproducible cryptographic example for `@webdecoy/node`. Generate a demo key,
sign a request, and inspect how verification changes when a covered URL is
modified, the signature expires, the signer is unknown, or no signature exists.

This is a Node.js console example, not a server or hosted browser UI.

```bash
cd playgrounds/signed-agent
npm ci --workspaces=false
npm run dev
npm test
```

Requires Node.js 20+. The standalone folder pins the published SDK and needs no
WebDecoy account, API key, or monorepo build.

| Controlled scenario | SDK status | Reason |
| --- | --- | --- |
| Signature matches the generated trusted key | `verified` | — |
| Covered target URL changed after signing | `impersonation` | `bad-signature` |
| Trusted-key signature is outside its validity window | `impersonation` | `expired` |
| Actual signature from a key absent from the trusted directory | `claimed` | `unknown-key` |
| Request carries no signature | `none` | — |

`proof.mjs` runs real Ed25519 signing and SDK verification. Only the trusted
**public-key directory response** is supplied by an injected local fixture;
the SDK verifier and its verdicts are not mocked. The demo private keys are
generated in memory, never printed or saved. No request to an AI provider is made.

The demo signs `@method`, `@target-uri`, and `signature-agent` with the
`web-bot-auth` tag. It sets zero clock tolerance to make the expired case explicit;
production SDK defaults differ. This fixture exercises the SDK's supported
signature profile, not all RFC 9421 variants or every agent implementation.

A verified signature is evidence of a trusted signing key and its covered
request components. Your authorization, rate limits and abuse policy still
apply. The SDK's `impersonation` label is a failed known-key verification result;
expiration or changed components can also come from clock or proxy issues and
do not alone establish malicious intent.

[Next.js integration and verdict guide](../../docs/verify-ai-agents-web-bot-auth.md)
· [Interactive Express tripwire playground](../express/README.md)
· [WebDecoy Node SDK](https://github.com/WebDecoy/node)
7 changes: 7 additions & 0 deletions playgrounds/signed-agent/demo.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import { runProof } from './proof.mjs';
const proof = await runProof();
console.log('WebDecoy signed-agent verification: controlled local fixture');
console.log(proof.fixture);
console.table(proof.results);
console.log(`Fixture directory reads: ${proof.directoryReads}. No API key supplied.`);
console.log('A verified signature establishes a trusted key, not permission to use your app.');
Loading
Loading