Skip to main content
ModPageSpeed 2.0: AVIF, WebP, and critical CSS — up to 69% less page weight on the live demo
2.0 1.1

Serve clean content to AI agents

Serve a rendered, readable copy of your pages to AI agents at the same URL — off by default, unlocked by license, with nothing leaving your server.

AI agents and LLM crawlers increasingly fetch your pages to read, summarize, and act on them — but a modern, JavaScript-rendered site often looks empty to a client that does not execute scripts. ModPageSpeed 2.0 can serve those agents a rendered, readable copy of each page, at the same URL, without changing anything a browser or search engine sees.

How it works

When an agent requests a page with Accept: text/markdown and your license carries the agent_optimize entitlement, ModPageSpeed 2.0 serves a Markdown representation produced from the fully-rendered DOM — JavaScript executed, layout settled. JS-heavy pages that are otherwise invisible to agents become clean, structured text. Everyone else gets your normal optimized HTML at that same URL.

The agent response is content-negotiated, with these guarantees:

  • Vary: Accept — shared caches never serve the Markdown copy to a browser, or the HTML to an agent.
  • X-Robots-Tag: noindex and Cache-Control: private — the agent copy stays out of search indexes and shared caches.
  • The page body and Content-Type for browsers are unchanged — standard content negotiation on Accept, not cloaking.

The rendering runs in the ModPageSpeed 2.0 worker process behind nginx, reusing the same headless-Chrome subsystem as browser analysis.

Off by default, unlocked by license

The capability is present in every ModPageSpeed 2.0 build, but it does nothing until two things are true: you enable agent optimization, and your license token carries the additive agent_optimize entitlement. The entitlement rides on the same license rail as everything else (a signed claim on your existing token) — no new token format, no separate download.

If the entitlement is absent or your license lapses, an agent request transparently receives the normal HTML of the same URL — never an error, never a half-rendered page. Turning the feature off is always safe.

Serve an /llms.txt site index

ModPageSpeed 2.0 can also publish a synthesized /llms.txt for your site — a compact Markdown index that points AI agents at your important pages, in the llmstxt.org format. The worker builds it from your own sitemap.xml (intersected with the paths you allow), with a one-line title and summary per page, and nginx serves it at /llms.txt with X-Robots-Tag: noindex, Cache-Control: private, and Content-Type: text/markdown.

It is off by default and gated by the same agent_optimize entitlement — one capability, one license rail. Per-page summaries come for free: from a page’s already-rendered agent copy when one exists, or from a cheap title/description read otherwise. The file is never built by rendering pages on demand. The index is generated and cached off the serving path and refreshed lazily when your sitemap changes (or once a day at most).

Content is left out if your robots.txt tells AI crawlers to stay away, or if a page opts out with noai / X-Robots-Tag: noai. A site that blocks AI crawlers entirely gets no /llms.txt at all.

Like the rendered-Markdown copy, /llms.txt ships in ModPageSpeed 2.0 only — it needs the worker and headless-Chrome substrate. mod_pagespeed 1.1 does not produce it.

Nothing leaves your server

Rendering, cleaning, caching, serving, and the license check all happen on your own server. Serving a cached response — a page, the agent Markdown copy, or /llms.txt — makes no third-party network call: no DNS, no telemetry, no analytics, no beacon.

The only network activity happens off the serving path:

  • Building the /llms.txt index fetches your own sitemap.xml and pages.
  • Rendering may fetch a page’s own resources.
  • Both go through the same SSRF-guarded fetcher, IP-pinned to your origin.
  • License renewal is a periodic background heartbeat — disable it for a fully air-gapped deployment.

This makes agent optimization suitable for sovereign and regulated environments where content must not leave the premises.

Enabling it

Agent optimization is a licensed capability. Enable agent optimization in your ModPageSpeed 2.0 configuration and ensure your license token includes the agent_optimize entitlement. To obtain an entitlement for your license, get in touch — we will scope it to your deployment.

See also: Analyze pages with headless Chrome and License activation.

Frequently asked questions

Does this change what browsers and search engines see?

No. Browsers and crawlers receive your normal HTML at the same URL. Only a request that explicitly negotiates the agent representation (Accept: text/markdown) and is licensed for it receives the rendered-Markdown copy. The response carries Vary: Accept so shared caches never mix the two.

Is it on by default?

No. The capability ships in every build but stays off until you enable it and your license carries the agent_optimize entitlement. The binary is byte-identical for licensed and unlicensed installs — there is no separate “pro” build.

Does my content leave my server?

No. Rendering, cleaning, caching, and serving all happen on your own server, and the license check is offline. Serving a cached response performs zero outbound network I/O.

Is this available in mod_pagespeed 1.1?

The rendered-Markdown variant is a ModPageSpeed 2.0 capability — it runs in the 2.0 worker behind nginx using headless Chrome. mod_pagespeed 1.1 recognizes and safely declines the agent request (it always serves normal HTML); it does not produce the rendered variant.

What is the synthesized /llms.txt, and is it your /llms.txt?

ModPageSpeed 2.0 can publish a synthesized /llms.txt for YOUR site — a Markdown index built from your own sitemap, off by default and gated by the same agent_optimize entitlement. It is unrelated to the /llms.txt on modpagespeed.com, which describes our product. The index is built off the serving path and serving it performs zero outbound network I/O. It is a 2.0-only capability; mod_pagespeed 1.1 does not produce it.