Skip to main content
mod_pagespeed 2.1 is here — open source (Apache-2.0): the module you know, plus the optimizer worker

Serve Markdown to AI agents

Serve AI agents a rendered Markdown copy of your pages at the same URL, and synthesize an /llms.txt — off by default, nothing leaves your server.

On this page

AI agents and LLM crawlers fetch your pages to read, summarize, and act on them — but a JavaScript-rendered site often looks empty to a client that does not execute scripts. The mod_pagespeed 2.1 optimizer worker can serve those agents a rendered, readable copy of each page, at the same URL, without changing anything a browser or search engine sees.

How it works

When an agent requests a page with Accept: text/markdown and you have enabled agent optimization, the optimizer worker serves a Markdown representation produced from the fully-rendered DOM — JavaScript executed, layout settled. JS-heavy pages that are otherwise invisible to agents become clean, structured text. Everyone else gets your normal optimized HTML at that same URL.

The agent response is content-negotiated, with these guarantees:

  • Vary: Accept — shared caches never serve the Markdown copy to a browser, or the HTML to an agent.
  • X-Robots-Tag: noindex and Cache-Control: private — the agent copy stays out of search indexes and shared caches.
  • The page body and Content-Type for browsers are unchanged — standard content negotiation on Accept, not cloaking.

The rendering runs in the optimizer worker process behind nginx, reusing the same headless-Chrome subsystem as browser analysis.

Off by default

The capability is present in every build, but it does nothing until you enable agent optimization on the worker (--agent-optimize, or PAGESPEED_AGENT_OPTIMIZE=true in the Docker images). It also requires browser analysis (--enable-browser-analysis), since the Markdown copy is produced from the rendered DOM.

While the flag is off, an agent request transparently receives the normal HTML of the same URL — never an error, never a half-rendered page. Turning the feature off is always safe.

Serve an /llms.txt site index

The optimizer worker can also publish a synthesized /llms.txt for your site — a compact Markdown index that points AI agents at your important pages, in the llmstxt.org format. The worker builds it from your own sitemap.xml (intersected with the paths you allow), with a one-line title and summary per page, and nginx serves it at /llms.txt with X-Robots-Tag: noindex, Cache-Control: private, and Content-Type: text/markdown.

It is off by default and enabled with its own flag (--agent-optimize-llms-txt, which requires --agent-optimize). Per-page summaries come for free: from a page’s already-rendered agent copy when one exists, or from a cheap title/description read otherwise. The file is never built by rendering pages on demand. The index is generated and cached off the serving path and refreshed lazily when your sitemap changes (or once a day at most).

Content is left out if your robots.txt tells AI crawlers to stay away, or if a page opts out with noai / X-Robots-Tag: noai. A site that blocks AI crawlers entirely gets no /llms.txt at all.

Like the rendered-Markdown copy, /llms.txt needs the optimizer worker and its headless-Chrome substrate; the module alone does not produce it.

Nothing leaves your server

Rendering, cleaning, caching, and serving all happen on your own server. Serving a cached response — a page, the agent Markdown copy, or /llms.txt — makes no third-party network call: no DNS, no telemetry, no analytics, no beacon.

The only network activity happens off the serving path:

  • Building the /llms.txt index fetches your own sitemap.xml and pages.
  • Rendering may fetch a page’s own resources.
  • Both go through the same SSRF-guarded fetcher, IP-pinned to your origin.

This makes agent optimization suitable for sovereign and regulated environments where content must not leave the premises.

Enabling it

Start the worker with --agent-optimize (and --enable-browser-analysis), then add --agent-optimize-llms-txt if you also want the synthesized /llms.txt. Evaluate it in staging first: the capability is experimental and its flags may change.

See also: Analyze pages with headless Chrome. For the design and rationale, read Serve Markdown to AI agents and the deep-dive on serving Markdown and a synthesized /llms.txt to AI crawlers.

Frequently asked questions

Does this change what browsers and search engines see?

No. Browsers and crawlers receive your normal HTML at the same URL. Only a request that explicitly negotiates the agent representation (Accept: text/markdown), on a deployment where you have enabled agent optimization, receives the rendered-Markdown copy. The response carries Vary: Accept so shared caches never mix the two.

Is it on by default?

No. The capability ships in every build but stays off until you enable it with the —agent-optimize worker flag. There is no separate build.

Does my content leave my server?

No. Rendering, cleaning, caching, and serving all happen on your own server. Serving a cached response performs zero outbound network I/O.

Does this work with the module alone, or does it need the optimizer worker?

Yes. The rendered-Markdown variant runs in the optimizer worker behind nginx, using headless Chrome. The module recognizes and safely declines the agent request (it always serves normal HTML); it does not produce the rendered variant.

What is the synthesized /llms.txt, and is it your /llms.txt?

The optimizer worker can publish a synthesized /llms.txt for YOUR site — a Markdown index built from your own sitemap, off by default and enabled by its own worker flag. It is unrelated to the /llms.txt on modpagespeed.com, which describes our product. The index is built off the serving path and serving it performs zero outbound network I/O. It needs the worker: the module alone does not produce it.