Tools
AI crawler cost meter
Drop an access log. Nothing from it leaves your browser. The page reports what each AI crawler asked for, what that cost at the rate you set, and how many visits AI assistants sent back.
Result
Per crawler (claimed user agent)
| Crawler | Operator | Kind | Requests | Share of requests | Page requests | Bytes | Share of bytes | Estimated cost |
|---|
Per operator: crawled against sent back
| Operator | Requests | Bytes | Crawler page requests | Visits sent back | Ratio |
|---|
What to do about it
Block AI crawlers in nginx, Apache or robots.txt
The ai.robots.txt project maintains a robots.txt, an nginx config and an .htaccess that block the crawlers on its list; the list on this page is a filtered subset of it.
Get the block files →Serve agents a clean copy instead
A page that answers Accept: text/markdown with Markdown is usually much smaller than its HTML and is readable by agents that run no JavaScript.
How to serve Markdown to agents →Know which crawler is really knocking
Web Bot Auth lets a crawler sign its requests. The mod_pagespeed module can verify those signatures, observe-only and off by default; the post explains what the verdict is and is not.
Verify AI crawlers with Web Bot Auth →Put a price on crawling at the origin
The post explains how a crawl price at the origin works, what it needs from the crawler, and where that stands.
Pay-per-crawl at the origin →For publishers and hosters with heavy AI crawler traffic
If AI crawlers are a line item for you, talk to us about measuring and shaping that traffic at the origin.
Talk to us →Check the claim yourself
- Load this page.
- Switch your network off: airplane mode, or your browser's offline switch in its developer tools.
- Drop your log. The result appears anyway.
Once the page has loaded, the only requests it makes are for the site's analytics and for this site's own files (a font, or the search index if you open site search), plus any form on the page you fill in yourself. The analytics script loads from umami.we-amp.com, and its events are posted to this site's own /api/send, which forwards them: a page view, one event when an analysis finishes, and a count when you click a link or a share button on this page. None of these carries anything from your log: no counts, no hosts, no paths. The parsing code is published at https://github.com/We-Amp/pagespeed-optimizer/tree/main/website/src/lib/crawler-cost. Browser extensions can read any page, and that is outside what a page can promise.
How this page counts
- Formats: nginx and Apache combined (prefixed variants and fields appended after the user agent included), Apache common (no user agent, so nothing can be classified), IIS W3C extended by its Fields line (a new Fields line mid-file applies from there). Gzip is recognised by its magic bytes, every member of a concatenated archive is read, and lines over 64 KB are skipped and counted.
- Bytes are the bytes your server logged: the response body for nginx and Apache, the whole response including headers for IIS (sc-bytes). Gigabytes are decimal (1 GB = 1,000,000,000 bytes).
- A crawler is recognised by the user agent it claims. Spoofing cuts both ways: an impersonator claiming a crawler's agent is counted as that crawler, and a crawler sending a browser agent is missed. Every per-crawler number is an estimate in both directions.
- A page request is a request whose path has no static asset extension (.html counts as a page). Crawler page requests count any status; visits sent back count only 200 and 304.
- A visit sent back is a page request from a browser whose Referer names an AI assistant, or whose query carries a known marker such as utm_source=chatgpt.com. Referrer policies strip many referers, so this is a floor. Answers inside Google or Bing search arrive as ordinary search referrers and are not counted.
- Fetches made on a person's behalf (kind "fetcher", for example ChatGPT-User) are listed apart and left out of the headline ratio.
- The headline ratio divides crawler page requests by visits sent back, so pages stand on both sides. All crawler requests, assets included, are in the totals.
- A log without user agents cannot show crawlers; the page says so instead of reporting zero.