Verify AI crawlers with Web Bot Auth

Any client can claim to be any crawler — a User-Agent string is unverified text. Web Bot Auth fixes that with cryptography: a bot signs its requests with an Ed25519 key (RFC 9421 HTTP Message Signatures) and publishes the public half in a JWKS key directory at a well-known HTTPS URL. ModPageSpeed 2.0 can verify those signatures at your origin and tell you, per request, which bot actually sent it.

What you get

For every request that carries Web Bot Auth signature material, the response gains an x-verified-bot header:

Request	x-verified-bot
Valid signature, key id in your verified-bots map	<bot-name>, ed25519-verified
Valid signature, key id not in the map	signed-agent
Tampered/expired signature, or an unknown key	unknown
No signature material (browsers, ordinary crawlers)	no header

Requests without signature headers are untouched — the label never inflates ordinary traffic. Aggregate counts are exported at /v1/metrics:

pagespeed_webbotauth_signed_requests_total{result="verified"}
pagespeed_webbotauth_signed_requests_total{result="invalid"}
pagespeed_webbotauth_signed_requests_total{result="other"}

verified covers both mapped and unmapped valid signatures; invalid is signature material that fail-closed; other is parseable RFC 9421 material that is not Web Bot Auth (for example a CDN signing scheme), which is treated as unsigned.

Enabling it

Three environment variables on the worker container (Docker and Helm deploys alike). Changing them requires a worker restart.

services:
  worker:
    image: ghcr.io/we-amp/pagespeed-worker:2.0.35
    environment:
      - PAGESPEED_WEB_BOT_AUTH=true
      # Comma-separated HTTPS JWKS key-directory URLs (no spaces).
      - PAGESPEED_WEB_BOT_AUTH_KEY_DIRECTORIES=https://example-bot.com/.well-known/http-message-signatures-directory
      # Optional: promote key ids to readable bot names ("keyid=name,...").
      - PAGESPEED_WEB_BOT_AUTH_VERIFIED_BOTS=NFcWBUXB-example-key-id=examplebot

The matching worker command-line flags are --web-bot-auth, --web-bot-auth-key-directory <url> (repeatable) and --web-bot-auth-verified-bots "keyid=name,...".

• Key directories must be HTTPS; the worker rejects plain-HTTP URLs at startup. The worker fetches and refreshes each directory periodically in the background, so key rotations by the bot operator are picked up without a restart — and verification itself never waits on the network.
• The verified-bots map is your allowlist of identities worth naming. A key id you have mapped renders as <name>, ed25519-verified; a valid signature from an unmapped key still verifies, but renders as the generic signed-agent.

Reading the results

Spot-check the header from your logs or by inspecting responses to a signed crawler you operate. For trend lines, scrape /v1/metrics (see the HTTP API for authentication) — the three counters make it easy to answer “how much of my AI-crawler traffic is actually verifiable?”

Unsigned bot traffic — the majority today — simply shows no header and no counter increment. The interesting signal is the split between verified (cryptographically attributable crawls) and invalid (something sent signature material that did not check out).