Web Bot Auth key directory probe, 19 October 2026

Data behind 55 of 150 AI crawler operators publish Web Bot Auth keys. One GET per target URL at 2026-10-19T18:30Z.

• probe.py: the probe. probe.py targets.csv > results.tsv. Needs Python 3 and curl.
• build_targets.py: builds the operator population and the target list from the two sources below, with the operator-to-domain mapping used.
• targets-excluded.csv: the entries left out, with the reason.
• results.tsv: 396 rows, one per URL, timestamped. Columns are documented in the probe's header comment.

Sources

• ai.robots.txt robots.json at commit 6abe630 (18 October 2026). Fetch it from that repository at that commit.
• Cloudflare Radar bots directory, categories AI crawler, AI assistant and AI search. The raw Radar dataset is not republished here; pull it from the Radar API (GET /client/v4/radar/bots) with your own token. Only the fields the builder reads (bot slug, operator, category, signature agent URL) feed the target list.

Rerun

python3 build_targets.py robots.json radar-details.json targets.csv
python3 probe.py targets.csv > results.tsv

Counts move; rerun before relying on any row.