Browser analysis with headless Chrome

ModPageSpeed 2.0 can render pages in headless Chrome instead of relying on heuristics alone. Browser analysis reads critical CSS from real CSS Coverage data, detects the true Largest Contentful Paint element, measures rendered image dimensions, and — for above-the-fold CSS — confirms in a real render that the page still looks the same before allowing its stylesheet to be deferred.

Browser analysis is strictly additive. Every failure falls back to the heuristic path, so pages still get optimized — just through the faster, less precise heuristic pipeline. The trade-off is accuracy for latency: a rendered profile is more precise than a heuristic critical-CSS estimate, but it costs a headless render the first time a page template is seen.

Enabling Browser Analysis

Browser analysis is off by default. Enable it with the --enable-browser-analysis flag and ensure Chrome (or chrome-headless-shell) is available in the container:

factory_worker \
  --cache-path /data/cache.vol \
  --enable-browser-analysis \
  --chrome-binary /usr/bin/chrome-headless-shell

The Docker release images (ghcr.io/we-amp/pagespeed-worker and the combined ghcr.io/we-amp/pagespeed-combined) ship with Chromium pre-installed. In a container, enable analysis by setting PAGESPEED_ENABLE_BROWSER_ANALYSIS=true — the entrypoint passes --enable-browser-analysis for you. No --shm-size tuning is needed: the worker runs Chrome with --disable-dev-shm-usage, so it does not rely on the container’s /dev/shm.

Architecture

Worker (libuv event loop)
  |
  +-- BrowserAnalysisManager
        |
        +-- AnalysisQueue        -- bounded priority queue with dedup
        |
        +-- ChromeProcess        -- spawn/recycle/RSS monitor
        |     |
        |     +-- CdpClient      -- JSON-RPC over pipe (FD 3/4)
        |
        +-- BrowserCssExtractor  -- CSS Coverage API -> critical CSS
        +-- PageAnalyzer         -- LCP, fold, CLS, image dims
        +-- UnusedCssRemover     -- dead rule removal
        +-- VisualRegressionGate -- PNG pixel diff validation
        +-- FontGlyphScanner     -- code point scanning + @font-face
        +-- ScriptCoverageAnalyzer -- Profiler coverage + deferral

BrowserAnalysisManager owns the Chrome lifecycle, analysis queue, and the CDP pipeline. It runs on the main libuv event loop (where CDP must operate). Worker thread pool threads enqueue analysis requests via uv_async_send().

CDP Pipe Transport

Chrome DevTools Protocol communication happens over --remote-debugging-pipe (file descriptors 3 and 4), not over a WebSocket. Messages are null-byte delimited JSON-RPC. This avoids the overhead and port management of the WebSocket debugging protocol.

Design decisions:

• Per-command uv_timer_t timeout (default 30s)
• Large CDP messages (>64KB) parsed off the event loop via uv_queue_work()
• CancelAll() on pipe EOF resolves all pending callbacks

How It Works

1. The worker thread runs HtmlScanner::Scan() to extract page structure
2. TemplateDetector::HashStructure() computes an FNV-1a hash of the DOM structure, identifying the page template
3. LookupProfile() checks the cache for an existing OptimizationProfile for this template hash
4. Profile found: browser-validated critical CSS and LCP data are used instead of heuristics
5. No profile: EnqueueAnalysis() sends the request to the main event loop via uv_async_send()
6. DrainQueue() dequeues items and runs the analysis pipeline across three viewports: Mobile (375x667), Tablet (768x1024), Desktop (1440x900)
7. The resulting OptimizationProfile is stored in the cache with SentinelId::kBrowserProfile

CSS Cache Inlining

Before passing HTML to Chrome, the worker resolves <link rel="stylesheet"> tags against the Cyclone cache and injects <style> blocks into the HTML. This enables Chrome’s CSS Coverage API to compute real coverage percentages instead of returning 0% for external stylesheets. The CSS cache inlining trick for the Coverage API explains why the raw external-stylesheet path reports 0% and how the inline step recovers accurate numbers.

Guards prevent abuse: 50 stylesheet cap, 2MB per-stylesheet cap, 10MB total HTML cap.

Analysis Components

Component	Purpose
BrowserCssExtractor	Uses Chrome’s CSS Coverage API to identify which CSS rules are actually used on each viewport. Produces per-viewport critical CSS.
PageAnalyzer	Detects the real LCP element, measures fold position, computes CLS, and reads rendered image dimensions.
UnusedCssRemover	Takes Coverage data and removes dead rules from stylesheets.
VisualRegressionGate	Renders two versions of a page and compares the above-the-fold region pixel-by-pixel. Drives the critical-CSS check below.
FontGlyphScanner	Scans the DOM with TreeWalker for code points used on the page. Maps them to @font-face declarations for future subsetting.
ScriptCoverageAnalyzer	Uses Chrome’s Profiler domain to measure JS code coverage. Identifies scripts safe to defer.

What the pixel comparison actually gates

One optimization is gated on it: deferring a stylesheet. During analysis, the page is rendered twice at each viewport — once with its whole stylesheet, once with only the above-the-fold block that would be inlined — and the two above-the-fold regions are compared. The stylesheet is made non-render-blocking only when they match. If the check does not run, or does not pass, the stylesheet stays render-blocking and the above-the-fold CSS is still inlined, so the page loses the deferral and keeps everything else.

The check is tied to the exact stylesheet it was made against, so publishing new CSS re-checks before deferring again, and a page whose analysis has not finished yet keeps its stylesheet render-blocking.

The other optimizations are not gated this way. Lazy-loading, image sizing, script deferral and the rest are applied on their own evidence — coverage data, measured dimensions, fold position — and are not pixel-compared before use.

Three things the check cannot see

The comparison render is deliberately sealed off from the network and from scripts, so that analysing a page can never be turned into a way of reaching something else. Two consequences follow, and neither is a setting:

• Scripts do not run. A fold whose layout is established by JavaScript is compared against a version of the page the visitor never sees.
• Images, web fonts and imported stylesheets do not load. They are absent from both renders, so a fold that depends on a background image or a @font-face looks the same in each, and the difference the visitor would see is not visible to the check.

The third is about which pages a passed check covers:

• The check is per template, not per page. Pages that share a template share one confirmation. The above-the-fold CSS is still worked out per page, but the confirmation that it covers the fold was made on whichever page of that template was analysed. A sibling page whose fold needs something that one did not can be deferred on its confirmation.

If a page’s above-the-fold appearance depends on any of these, verify it yourself before relying on stylesheet deferral there — or turn deferral off for that site with --no-async-css. In particular, seeing a flash on one page of a template is a reason to turn deferral off for the site, not to expect the check to have caught it.

Note also that pages whose above-the-fold CSS already accounts for most of their stylesheet are not deferred at all — inlining most of a sheet and then downloading it again is a loss — so those pages are never checked, because there is nothing to authorize.

Script Coverage Analysis

When browser analysis is enabled, the ScriptCoverageAnalyzer component uses Chrome’s Profiler domain to measure JavaScript code coverage. This identifies scripts that are safe to defer, improving page load performance by reducing parser-blocking JavaScript. The same Coverage-API technique drives removing unused JavaScript with Chrome coverage.

How It Works

1. The analyzer loads the page with JavaScript enabled (Profiler + Coverage APIs)
2. Each external script’s coverage is measured during page load
3. Scripts are classified into deferral categories based on coverage data and execution timing

Deferral Categories

Category	Description
kSafeToDefer	Script has low main-thread impact; safe to add defer
kCandidateForAsync	Script is independent; could use async instead
kAlreadyAsync	Script already has async or defer attribute
kKeepSynchronous	Script must execute synchronously (DOM-dependent, inline handlers)

SSRF Defense

Script analysis enables JavaScript execution in Chrome (required for accurate coverage measurement). The other three SSRF defense layers remain active: network offline mode, Fetch interception, and DNS-level blocking. Chrome cannot make outbound connections even with JavaScript enabled.

Configuration

Flag	Default	Description
--no-browser-script-analysis	(enabled)	Disable script coverage analysis

Script analysis results feed into the optimization policy engine, which decides whether to enable script deferral for each URL template.

Optimization Policy

The optimization policy engine computes per-template decisions about optional HTML transforms based on browser analysis data. It runs after profile generation and stores the policy alongside the optimization profile in cache.

Policy Fields

Field	Condition	Description
async_css_enabled	Avg CSS coverage < 50%	Advises that async loading is worth considering for this template. Advisory only — it does not enable deferral. Deferral additionally requires a confirmed above-the-fold result for the page, bound to the stylesheet being served.
script_deferral_enabled	Deferrable scripts detected	Enable defer attribute on safe scripts

Stats Counters

Counter	Description
policy.computed	Total optimization policies computed
policy.async_css_enabled	Times async CSS was enabled by policy
policy.async_css_suppressed_low_coverage	Times stylesheet deferral was refused because the inlined above-the-fold CSS was too thin to bridge first paint
policy.async_css_suppressed_unvalidated	Times stylesheet deferral was refused because the page has no confirmed above-the-fold result bound to the stylesheet being served (not analyzed yet, or the stylesheet changed since)
policy.async_css_record_dropped_empty_derivation	Times a page’s confirmed above-the-fold result was set aside because no above-the-fold CSS could be measured for that specific page, so the confirmation does not describe what would be inlined
policy.script_deferral_enabled	Times script deferral was enabled by policy

These counters appear in /v1/stats JSON, /v1/metrics Prometheus output, the management socket STATS command, and the web console metrics page.

Chrome Process Management

Lifecycle

1. ChromeProcess::Start() spawns Chrome with headless flags and pipe transport
2. CDP commands flow through CdpClient for page analysis
3. After each page, IncrementPageCount() checks the recycle threshold
4. At the recycle threshold, Stop() sends SIGTERM (then SIGKILL after 5s)
5. A fresh Chrome process starts for the next batch

Launch Flags

Chrome is spawned with strict isolation flags:

• --headless=new — new headless mode
• --remote-debugging-pipe — FD 3/4 pipe transport
• --disable-gpu — no GPU required
• --no-sandbox — required in containers (Chrome must run with minimal container privileges)
• --host-resolver-rules="MAP * ~NOTFOUND" — DNS-level SSRF block
• --disable-dev-shm-usage — avoids /dev/shm exhaustion in containers
• Various isolation flags (--disable-extensions, --disable-background-networking, --no-first-run, etc.)

RSS Monitoring

On Linux, the worker reads /proc/pid/status VmRSS every 5 seconds. When Chrome exceeds --chrome-max-memory (default 512MB), the worker stops it and starts a fresh instance. This prevents memory leaks from accumulating across hundreds of pages.

SSRF Defense (4 Layers)

Browser analysis operates on cached content, not live network requests. Four layers prevent Chrome from making any outbound connections (the reasoning behind this air-gapped design is covered in air-gapped headless fetch and SSRF pinning):

1. Network.emulateNetworkConditions({offline: true}) — blocks all network
2. Fetch.enable + Fetch.requestPaused — intercept and fail all requests
3. Emulation.setScriptExecutionDisabled({value: true}) — no JS execution (CSS extractor and visual regression gate)
4. --host-resolver-rules="MAP * ~NOTFOUND" — Chrome-level DNS block

Font Glyph Scanner and Script Coverage Analyzer enable JavaScript (they need it for accurate analysis) but still enforce the other three layers.

Configuration Flags

Flag	Default	Description
--enable-browser-analysis	off	Enable browser analysis pipeline
--chrome-binary	/usr/bin/chrome-headless-shell	Path to Chrome binary
--chrome-recycle-interval	100	Pages per Chrome instance before restart
--chrome-page-timeout	60000	Per-page analysis timeout in ms
--chrome-max-memory	512	Max Chrome RSS in MB before forced restart
--chrome-startup-timeout	10000	Chrome startup timeout in ms
--browser-queue-size	1000	Max queued analysis requests
--browser-profile-ttl	86400	Profile cache lifetime in seconds (24h)
--no-browser-critical-css	(enabled)	Disable browser-based critical CSS
--no-browser-lazy-loading	(enabled)	Disable browser-based lazy load decisions
--no-browser-lcp-preload	(enabled)	Disable browser-based LCP detection
--no-browser-image-sizing	(enabled)	Disable browser-based image dimensions
--no-browser-script-analysis	(enabled)	Disable script coverage analysis

All flags are also hot-reloadable via PATCH /v1/config from the web console.

Monitoring

Stats Counters

Browser analysis stats appear in the management socket STATS and BROWSER-STATUS commands, and in the web console dashboard:

Counter	Description
browser.profiles_generated	Templates analyzed and cached
browser.profiles_used	Cache hits on existing profiles
browser.analysis_errors	Failures (timeout, Chrome crash, etc.)
browser.chrome_crashes	Chrome process crashes
browser.queue_depth	Current queue size
browser.scripts_analyzed	Scripts evaluated by browser analysis
browser.scripts_deferrable	Scripts identified as safe to defer
browser.css_inlining_attempted	CSS inlining attempts
browser.css_inlining_stylesheets_cached	Stylesheets found in cache
browser.css_inlining_bytes_inlined	Total CSS bytes injected

Management Socket

The BROWSER-STATUS command on the management socket returns detailed JSON including Chrome state, queue contents, and per-profile statistics:

echo "BROWSER-STATUS" | socat - UNIX-CONNECT:/data/pagespeed.sock.mgmt

Error Handling

Every failure falls back to the heuristic path:

Failure	Behavior
Chrome binary not found	Heuristic only, no retry
Chrome fails to start	Retry after 2 seconds
Chrome crashes mid-analysis	Cancel current item, restart Chrome after 2s
Analysis timeout	Skip item, process next in queue
Cache read failure	Skip item
Queue full	Head-drop oldest item

The worker logs all browser analysis errors at the warning level. Monitor them in the debug console (/logs) or via the management socket.

Troubleshooting

Chrome not available (503 errors in waterfall/diff)

The web console’s waterfall viewer and visual diff features return 503 when Chrome is not running. Check:

1. Is --enable-browser-analysis set?
2. Does the Chrome binary exist at the configured path?
3. In Docker: is the worker image the full variant (not the minimal image)?

High chrome_crashes count

Frequent Chrome crashes usually indicate memory pressure:

• Lower --chrome-recycle-interval to restart Chrome more often
• Lower --chrome-max-memory to catch leaks earlier
• Check container memory limits — Chrome needs at least 256MB headroom

Profiles not being generated

If profiles_generated stays at zero while traffic flows:

1. Check queue_depth — if it stays at 0, analysis requests are not being enqueued. Verify --enable-browser-analysis is set.
2. Check analysis_errors — errors during analysis prevent profile creation.
3. Check css_inlining_stylesheets_cached — if external CSS is not yet cached, the worker waits for it before running browser analysis.

Visual Regression Gate false positives

The visual regression gate disables JavaScript (SSRF defense). Pages that rely on CSS-in-JS frameworks (styled-components, Emotion, etc.) will show differences because their styles are injected by JavaScript. This is a known limitation. The heuristic path optimizes these pages correctly.

Next Steps

• Web Console — Use the waterfall viewer and visual diff tools powered by browser analysis
• HTTP API Reference — BROWSER-STATUS management command and /v1/stats browser counters
• Configuration Reference — All browser analysis flags
• Troubleshooting — Chrome not found, CDP failures, and analysis timeout diagnostics
• AI readability scanner — a free hosted tool that renders any URL in headless Chromium and compares its raw HTML against the rendered DOM, showing what a non-JavaScript crawler sees