Page Speed Optimization Libraries  1.7.30.3
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines
Public Types | Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Friends
net_instaweb::RewriteDriver Class Reference

#include "rewrite_driver.h"

List of all members.

Public Types

enum  CssResolutionStatus { kWriteFailed, kNoResolutionNeeded, kSuccess }
 Status return-code for ResolveCssUrls.
enum  WaitMode { kNoWait, kWaitForCompletion, kWaitForCachedRender, kWaitForShutDown }
 Mode for BoundedWaitForCompletion. More...
enum  XhtmlStatus { kXhtmlUnknown, kIsXhtml, kIsNotXhtml }

Public Member Functions

 RewriteDriver (MessageHandler *message_handler, FileSystem *file_system, UrlAsyncFetcher *url_async_fetcher)
virtual ~RewriteDriver ()
RewriteDriverClone ()
void Clear ()
void SetServerContext (ServerContext *server_context)
bool MayCacheExtendCss () const
bool MayCacheExtendImages () const
bool MayCacheExtendPdfs () const
bool MayCacheExtendScripts () const
const GoogleString & user_agent () const
void SetUserAgent (const StringPiece &user_agent_string)
const RequestPropertiesrequest_properties () const
void ClearRequestProperties ()
 Reinitializes request_properties_, clearing any cached values.
bool using_spdy () const
 Returns true if the request we're rewriting was made using SPDY.
bool write_property_cache_dom_cohort () const
void set_write_property_cache_dom_cohort (bool x)
RequestContextPtr request_context ()
void set_request_context (const RequestContextPtr &x)
RequestTrace * trace_context ()
void TracePrintf (const char *fmt,...)
ResponseHeaders * mutable_response_headers ()
const ResponseHeaders * response_headers ()
void set_response_headers_ptr (ResponseHeaders *headers)
void SetRequestHeaders (const RequestHeaders &headers)
const RequestHeaders * request_headers () const
UserAgentMatcher * user_agent_matcher () const
void AddFilters ()
void AddOwnedEarlyPreRenderFilter (HtmlFilter *filter)
void PrependOwnedPreRenderFilter (HtmlFilter *filter)
 Adds a filter to the beginning of the pre-render chain, taking ownership.
void AppendOwnedPreRenderFilter (HtmlFilter *filter)
 Adds a filter to the end of the pre-render chain, taking ownership.
void AddOwnedPostRenderFilter (HtmlFilter *filter)
 Adds a filter to the end of the post-render chain, taking ownership.
void AddUnownedPostRenderFilter (HtmlFilter *filter)
 Same, without taking ownership.
void AppendRewriteFilter (RewriteFilter *filter)
void PrependRewriteFilter (RewriteFilter *filter)
void SetWriter (Writer *writer)
Writer * writer () const
bool FetchResource (const StringPiece &url, AsyncFetch *fetch)
void FetchInPlaceResource (const GoogleUrl &gurl, bool proxy_mode, AsyncFetch *async_fetch)
bool FetchOutputResource (const OutputResourcePtr &output_resource, RewriteFilter *filter, AsyncFetch *async_fetch)
OutputResourcePtr DecodeOutputResource (const GoogleUrl &url, RewriteFilter **filter) const
bool DecodeOutputResourceName (const GoogleUrl &url, ResourceNamer *name_out, OutputResourceKind *kind_out, RewriteFilter **filter_out) const
bool DecodeUrl (const GoogleUrl &url, StringVector *decoded_urls) const
 Decodes the incoming pagespeed url to original url(s).
FileSystem * file_system ()
UrlAsyncFetcherasync_fetcher ()
void SetSessionFetcher (UrlAsyncFetcher *f)
UrlAsyncFetcherdistributed_fetcher ()
void set_distributed_fetcher (UrlAsyncFetcher *fetcher)
 Does not take ownership.
CacheUrlAsyncFetcherCreateCacheFetcher ()
CacheUrlAsyncFetcherCreateCacheOnlyFetcher ()
 Returns a cache fetcher that does not fall back to an actual fetcher.
ServerContextserver_context () const
Statistics * statistics () const
AddInstrumentationFilteradd_instrumentation_filter ()
void set_custom_options (RewriteOptions *options)
 Takes ownership of 'options'.
void set_options_for_pool (RewriteDriverPool *pool, RewriteOptions *options)
RewriteDriverPoolcontrolling_pool ()
 Pool in which this driver can be recycled. May be NULL.
const RewriteOptionsoptions () const
 Return the options used for this RewriteDriver.
virtual bool StartParseId (const StringPiece &url, const StringPiece &id, const ContentType &content_type)
virtual void FinishParse ()
void FinishParseAsync (Function *callback)
void InfoAt (const RewriteContext *context, const char *msg,...) INSTAWEB_PRINTF_FORMAT(3
void OutputResourcePtr CreateOutputResourceFromResource (const StringPiece &filter_id, const UrlSegmentEncoder *encoder, const ResourceContext *data, const ResourcePtr &input_resource, OutputResourceKind kind)
OutputResourcePtr CreateOutputResourceWithPath (const StringPiece &mapped_path, const StringPiece &unmapped_path, const StringPiece &base_url, const StringPiece &filter_id, const StringPiece &name, OutputResourceKind kind)
void PopulateResourceNamer (const StringPiece &filter_id, const StringPiece &name, ResourceNamer *full_name)
OutputResourcePtr CreateOutputResourceWithUnmappedUrl (const GoogleUrl &unmapped_gurl, const StringPiece &filter_id, const StringPiece &name, OutputResourceKind kind)
OutputResourcePtr CreateOutputResourceWithMappedPath (const StringPiece &mapped_path, const StringPiece &unmapped_path, const StringPiece &filter_id, const StringPiece &name, OutputResourceKind kind)
OutputResourcePtr CreateOutputResourceWithPath (const StringPiece &path, const StringPiece &filter_id, const StringPiece &name, OutputResourceKind kind)
ResourcePtr CreateInputResource (const GoogleUrl &input_url)
ResourcePtr CreateInputResource (const GoogleUrl &input_url, bool allow_unauthorized_domain)
ResourcePtr CreateInputResourceAbsoluteUnchecked (const StringPiece &absolute_url)
bool MatchesBaseUrl (const GoogleUrl &input_url) const
bool MayRewriteUrl (const GoogleUrl &domain_url, const GoogleUrl &input_url, bool allow_unauthorized_domain, bool *is_authorized_domain) const
const GoogleUrl & base_url () const
StringPiece fetch_url () const
 The URL that was requested if FetchResource was called.
const GoogleUrl & decoded_base_url () const
StringPiece decoded_base () const
bool IsHttps () const
 Quick way to tell if the document url is https (ie was fetched via https).
const UrlSegmentEncoder * default_encoder () const
RewriteFilterFindFilter (const StringPiece &id) const
 Finds a filter with the given ID, or returns NULL if none found.
bool refs_before_base ()
 Returns refs_before_base.
void set_refs_before_base ()
StringPiece containing_charset ()
void set_containing_charset (const StringPiece charset)
HtmlResourceSlotPtr GetSlot (const ResourcePtr &resource, HtmlElement *elt, HtmlElement::Attribute *attr)
 Establishes a HtmlElement slot for rewriting.
bool InitiateRewrite (RewriteContext *rewrite_context)
void InitiateFetch (RewriteContext *rewrite_context)
void RewriteComplete (RewriteContext *rewrite_context, bool permit_render)
void ReportSlowRewrites (int num)
void Cleanup ()
void AddUserReference ()
GoogleString ToString (bool show_detached_contexts)
 Debugging routines to print out data about the driver.
void PrintState (bool show_detached_contexts)
 For debugging.
void PrintStateToErrorLog (bool show_detached_contexts)
 For logs.
void WaitForCompletion ()
void WaitForShutDown ()
void BoundedWaitFor (WaitMode mode, int64 timeout_ms)
void set_fully_rewrite_on_flush (bool x)
bool fully_rewrite_on_flush () const
 Returns if this response has a blocking rewrite or not.
void set_fast_blocking_rewrite (bool x)
bool fast_blocking_rewrite () const
void EnableBlockingRewrite (RequestHeaders *request_headers)
void set_externally_managed (bool x)
void DetachFetch ()
void DetachedFetchComplete ()
void FetchComplete ()
void DeleteRewriteContext (RewriteContext *rewrite_context)
int rewrite_deadline_ms ()
void set_max_page_processing_delay_ms (int x)
int max_page_processing_delay_ms ()
void set_device_type (UserAgentMatcher::DeviceType x)
 Sets the device type chosen for the current property_page.
UserAgentMatcher::DeviceType device_type ()
RewriteContextRegisterForPartitionKey (const GoogleString &partition_key, RewriteContext *candidate)
void DeregisterForPartitionKey (const GoogleString &partition_key, RewriteContext *candidate)
void RequestFlush ()
bool flush_requested () const
void ExecuteFlushIfRequested ()
void ExecuteFlushIfRequestedAsync (Function *callback)
virtual void Flush ()
void FlushAsync (Function *done)
void AddRewriteTask (Function *task)
 Queues up a task to run on the (high-priority) rewrite thread.
void AddLowPriorityRewriteTask (Function *task)
QueuedWorkerPool::Sequence * html_worker ()
QueuedWorkerPool::Sequence * rewrite_worker ()
QueuedWorkerPool::Sequence * low_priority_rewrite_worker ()
Scheduler * scheduler ()
DomainRewriteFilterdomain_rewriter ()
UrlLeftTrimFilterurl_trim_filter ()
CssResolutionStatus ResolveCssUrls (const GoogleUrl &input_css_base, const StringPiece &output_css_base, const StringPiece &contents, Writer *writer, MessageHandler *handler)
bool ShouldAbsolutifyUrl (const GoogleUrl &input_base, const GoogleUrl &output_base, bool *proxy_mode) const
void UpdatePropertyValueInDomCohort (AbstractPropertyPage *page, StringPiece property_name, StringPiece property_value)
PropertyPageproperty_page () const
FallbackPropertyPagefallback_property_page () const
void set_property_page (PropertyPage *page)
 Takes ownership of page.
void set_fallback_property_page (FallbackPropertyPage *page)
 Takes ownership of page.
void set_unowned_fallback_property_page (FallbackPropertyPage *page)
 Does not take the ownership of the page.
const CriticalLineInfo * critical_line_info () const
 Used by ImageRewriteFilter for identifying critical images.
void set_critical_line_info (CriticalLineInfo *critical_line_info)
const SplitHtmlConfigsplit_html_config ()
CriticalCssResult * critical_css_result () const
void set_critical_css_result (CriticalCssResult *critical_css_rules)
CriticalImagesInfocritical_images_info () const
 Used by ImageRewriteFilter for identifying critical images.
CriticalSelectorInfocritical_selector_info ()
void set_critical_selector_info (CriticalSelectorInfo *info)
void set_critical_images_info (CriticalImagesInfo *critical_images_info)
bool CriticalSelectorsEnabled () const
bool FlattenCssImportsEnabled () const
int num_inline_preview_images () const
void increment_num_inline_preview_images ()
 We expect to this method to be called on the HTML parser thread.
int num_flushed_early_pagespeed_resources () const
void increment_num_flushed_early_pagespeed_resources ()
void increment_async_events_count ()
void decrement_async_events_count ()
 Decrements a reference count bumped up by increment_async_events_count()
XhtmlStatus MimeTypeXhtmlStatus ()
void set_flushed_cached_html (bool x)
bool flushed_cached_html ()
void set_flushing_cached_html (bool x)
bool flushing_cached_html () const
void set_flushed_early (bool x)
bool flushed_early () const
void set_flushing_early (bool x)
bool flushing_early () const
void set_is_lazyload_script_flushed (bool x)
bool is_lazyload_script_flushed () const
FlushEarlyInfo * flush_early_info ()
 This method is not thread-safe. Call it only from the html parser thread.
FlushEarlyRenderInfo * flush_early_render_info () const
void set_flush_early_render_info (FlushEarlyRenderInfo *flush_early_render_info)
bool DebugMode () const
void SaveOriginalHeaders (const ResponseHeaders &response_headers)
AbstractLogRecordlog_record ()
DomStatsFilterdom_stats_filter () const
bool can_rewrite_resources () const
bool is_nested () const
 Determine whether this driver is nested inside another.
bool MetadataRequested (const RequestHeaders &request_headers) const
bool tried_to_distribute_fetch () const
 Did the driver attempt to distribute the fetch?
bool Write (const ResourceVector &inputs, const StringPiece &contents, const ContentType *type, StringPiece charset, OutputResource *output)
void set_defer_instrumentation_script (bool x)
bool defer_instrumentation_script () const

Static Public Member Functions

static void InitStats (Statistics *statistics)
 Initialize statistics for all filters that need it.
static void Initialize ()
 Initialize statics. Initialize/Terminate calls must be paired.
static void Terminate ()

Static Public Attributes

static const char * kPassThroughRequestAttributes [7]
static const char kDomCohort []
static const char kBeaconCohort []
 The cohort for properties that are written by the beacon handler.
static const char kLastRequestTimestamp []
static const char kParseSizeLimitExceeded []
 Tracks if we exceeded the maximum size limit of html which we should parse.
static const char kSubresourcesPropertyName []
 Flush Subresources Info associted with the HTML page.
static const char kStatusCodePropertyName []
 Status codes of previous responses.

Protected Member Functions

virtual void DetermineEnabledFiltersImpl ()

Friends

class DistributedRewriteContextTest
class RewriteContext
class RewriteDriverTest
class RewriteTestBase
class ServerContextTest
class ScanFilter
class CategorizedRefcount< RewriteDriver, RefCategory >

Detailed Description

This extends class HtmlParse (which should renamed HtmlContext) by providing context for rewriting resources (css, js, images).


Member Enumeration Documentation

Mode for BoundedWaitForCompletion.

Enumerator:
kNoWait 

Used internally. Do not pass in.

kWaitForCompletion 

wait for everything to complete (up to deadline)

kWaitForCachedRender 

wait for at least cached rewrites to complete, and anything else that finishes within deadline.

kWaitForShutDown 

Makes sure that all work, including any that's being done in background, finishes.

Indicates document's mimetype as XHTML, HTML, or is not known/something else. Note that in Apache we might not know the correct mimetype because a downstream module might change it. It's not clear how likely this is, since mod_rewrite and mod_mime run upstream of mod_pagespeed. However if anyone sets mimetype via "Header Add", it would affect the Browser's view of the document's mimetype (which is what determines the parsing) but mod_pagespeed would not know.

Note that we also have doctype().IsXhtml() but that indicates quirks-mode for CSS, and does not control how the parser parses the document.


Constructor & Destructor Documentation

Need explicit destructors to allow destruction of scoped_ptr-controlled instances without propagating the include files.


Member Function Documentation

Adds the filters from the options, specified by name in enabled_filters. This must be called explicitly after object construction to provide an opportunity to programatically add custom filters beyond those defined in RewriteOptions, via AddFilter(HtmlFilter* filter) (below).

Queues up a task to run on the low-priority rewrite thread. Such tasks are expected to be safely cancelable.

Adds a filter to the very beginning of the pre-render chain, taking ownership. This should only be used for filters that must run before any filter added via PrependOwnedPreRenderFilter.

Adds an extra external reference to the object. You should not normally need to call it (NewRewriteDriver does it initially), unless for some reason you want to pin the object (e.g. in tests). Matches up with Cleanup.

Add a RewriteFilter to the end of the pre-render chain and take ownership of the filter. This differs from AppendOwnedPreRenderFilter in that it adds the filter's ID into a dispatch table for serving rewritten resources. E.g. if your filter->id == "xy" and FetchResource("NAME.pagespeed.xy.HASH.EXT"...) is called, then RewriteDriver will dispatch to filter->Fetch().

This is used when the filter being added is not part of the core set built into RewriteDriver and RewriteOptions, such as platform-specific or server-specific filters, or filters invented for unit-testing the framework.

const GoogleUrl& net_instaweb::RewriteDriver::base_url ( ) const [inline]

Returns the appropriate base gurl to be used for resolving hrefs in the document. Note that HtmlParse::google_url() is the URL for the HTML file and is used for printing html syntax errors.

void net_instaweb::RewriteDriver::BoundedWaitFor ( WaitMode  mode,
int64  timeout_ms 
)

As above, but with a time bound, and taking a mode parameter to decide between WaitForCompletion or WaitForShutDown behavior. If timeout_ms <= 0, no time bound will be used.

Determines whether the system is healthy enough to rewrite resources. Currently, systems get sick based on the health of the metadata cache.

If there are not outstanding references to this RewriteDriver, delete it or recycle it to a free pool in the ServerContext. If this is a fetch, calling this also signals to the system that you are no longer interested in its results.

Clears the current request cache of resources and base URL. The filter-chain is left intact so that a new request can be issued. Deletes all RewriteContexts.

WaitForCompletion must be called prior to Clear().

Returns a fresh instance using the same options we do, using the same log record. Drivers should only be cloned within the same request.

Get/set the charset of the containing HTML page. See scan_filter.cc for an explanation of how this is determined, but NOTE that the determined charset can change as more of the HTML is seen, in particular after a meta tag.

Creates a cache fetcher that uses the driver's fetcher and its options. Note: this means the driver's fetcher must survive as long as this does.

ResourcePtr net_instaweb::RewriteDriver::CreateInputResource ( const GoogleUrl &  input_url)

Creates an input resource based on input_url. Returns NULL if the input resource url isn't valid, or can't legally be rewritten in the context of this page.

ResourcePtr net_instaweb::RewriteDriver::CreateInputResource ( const GoogleUrl &  input_url,
bool  allow_unauthorized_domain 
)

Creates an input resource paying attention to whether or not resources from unauthorized domains are to be allowed or not. Returns NULL if the input resource url isn't valid, or can't legally be rewritten in the context of this page (which could mean that it was a resource from an unauthorized domain being processed by a filter that does not allow unauthorized resources).

ResourcePtr net_instaweb::RewriteDriver::CreateInputResourceAbsoluteUnchecked ( const StringPiece &  absolute_url)

Creates an input resource from the given absolute url. Requires that the provided url has been checked, and can legally be rewritten in the current page context.

void OutputResourcePtr net_instaweb::RewriteDriver::CreateOutputResourceFromResource ( const StringPiece &  filter_id,
const UrlSegmentEncoder *  encoder,
const ResourceContext *  data,
const ResourcePtr &  input_resource,
OutputResourceKind  kind 
)

Creates a reference-counted pointer to a new OutputResource object.

The content type is taken from the input_resource, but can be modified with SetType later if that is not correct (e.g. due to image transcoding). Constructs an output resource corresponding to the specified input resource and encoded using the provided encoder. Assumes permissions checking occurred when the input resource was constructed, and does not do it again. To avoid if-chains, tolerates a NULL input_resource (by returning NULL).

Todo:
TODO(jmaessen, jmarantz): Do we want to permit NULL input_resources here? jmarantz has evinced a distaste.
OutputResourcePtr net_instaweb::RewriteDriver::CreateOutputResourceWithMappedPath ( const StringPiece &  mapped_path,
const StringPiece &  unmapped_path,
const StringPiece &  filter_id,
const StringPiece &  name,
OutputResourceKind  kind 
) [inline]

Version of CreateOutputResourceWithPath where the unmapped and mapped paths are different and the base_url is this driver's base_url.

OutputResourcePtr net_instaweb::RewriteDriver::CreateOutputResourceWithPath ( const StringPiece &  mapped_path,
const StringPiece &  unmapped_path,
const StringPiece &  base_url,
const StringPiece &  filter_id,
const StringPiece &  name,
OutputResourceKind  kind 
)

Creates an output resource where the name is provided. The intent is to be able to derive the content from the name, for example, by encoding URLs and metadata.

This method succeeds unless the filename is too long.

This name is prepended with path for writing hrefs, and the resulting url is encoded and stored at file_prefix when working with the file system. So hrefs are: /usr/bin:/usr/local/google/home/huibao/bin:/usr/lib/lightdm/lightdm:/usr/lib/google-golang/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin/.pagespeed[.$EXPERIMENT].. .

EXPERIMENT is set only when there is an active experiment_spec.

Could be private since you should use one of the versions below but put here with the rest like it and for documentation clarity.

OutputResourcePtr net_instaweb::RewriteDriver::CreateOutputResourceWithPath ( const StringPiece &  path,
const StringPiece &  filter_id,
const StringPiece &  name,
OutputResourceKind  kind 
) [inline]

Version of CreateOutputResourceWithPath where the unmapped and mapped paths and the base url are all the same. FOR TESTS ONLY.

OutputResourcePtr net_instaweb::RewriteDriver::CreateOutputResourceWithUnmappedUrl ( const GoogleUrl &  unmapped_gurl,
const StringPiece &  filter_id,
const StringPiece &  name,
OutputResourceKind  kind 
)

Version of CreateOutputResourceWithPath which first takes only the unmapped path and finds the mapped path using the DomainLawyer and the base_url is this driver's base_url.

This should only be called by the CriticalSelectorFinder. Normal users should call CriticalSelectorFinder::IsCriticalImage.

Todo:
TODO(jud): Remove when the finders reside in RewriteDriver and manage their own state.

Return true if we must prioritize critical selectors, and we should therefore enable its prerequisite filters as well.

bool net_instaweb::RewriteDriver::DebugMode ( ) const [inline]

Determines whether we are currently in Debug mode; meaning that the site owner or user has enabled filter kDebug.

const GoogleUrl& net_instaweb::RewriteDriver::decoded_base_url ( ) const [inline]

Returns the decoded version of base_gurl() in case it was encoded by a non-default UrlNamer (for the default UrlNamer this returns the same value as base_url()). Required when fetching a resource by its encoded name.

OutputResourcePtr net_instaweb::RewriteDriver::DecodeOutputResource ( const GoogleUrl &  url,
RewriteFilter **  filter 
) const

Attempts to decode an output resource based on the URL pattern without actually rewriting it. No permission checks are performed on the url, though it is parsed to see if it looks like the url of a generated resource (which should mean checking the hash to ensure we generated it ourselves).

Todo:
TODO(jmaessen): add url hash & check thereof.
bool net_instaweb::RewriteDriver::DecodeOutputResourceName ( const GoogleUrl &  url,
ResourceNamer name_out,
OutputResourceKind kind_out,
RewriteFilter **  filter_out 
) const

As above, but does not actually create a resource object, and instead outputs the decoded information into the various out parameters. Returns whether decoding successful or not.

Deletes the specified RewriteContext. If this is the last RewriteContext active on this Driver, and there is no other outstanding activity, then the RewriteDriver itself can be recycled, and WaitForCompletion can return.

We expect to this method to be called on the Rewrite thread.

void net_instaweb::RewriteDriver::DeregisterForPartitionKey ( const GoogleString &  partition_key,
RewriteContext candidate 
)

Must be called after all other rewrites that are currently relying on this one have had their RepeatedSuccess or RepeatedFailure methods called.

Must only be called from rewrite thread.

Called by RewriteContext when a detached async fetch is complete, allowing the RewriteDriver to be recycled if FetchComplete() got invoked as well.

Called by RewriteContext to let RewriteDriver know it will be continuing on the fetch in background, and so it should defer doing full cleanup sequences until DetachedFetchComplete() is called.

Used by CacheExtender, CssCombineFilter, etc. for rewriting domains of sub-resources in CSS.

void net_instaweb::RewriteDriver::EnableBlockingRewrite ( RequestHeaders *  request_headers)

If the value of X-PSA-Blocking-Rewrite request header matches the blocking rewrite key, set fully_rewrite_on_flush flag.

Executes an Flush() if RequestFlush() was called, e.g. from the Listener Filter (see set_event_listener below). Consider an HTML parse driven by a UrlAsyncFetcher. When the UrlAsyncFetcher temporarily runs out of bytes to read, it calls response_writer->Flush(). When that happens, we may want to consider flushing the outstanding HTML events through the system so that the browser can start fetching subresources and rendering. The event_listener (see set_event_listener below) helps determine whether enough "interesting" events have passed in the current flush window so that we should take this incoming network pause as an opportunity.

Asynchronous version of the above. Note that you should not attempt to write out any data until the callback is invoked. (If a flush is not needed, the callback will be invoked immediately).

Returns the property page which contains the cached properties associated with the current URL and fallback URL (i.e. without query params). This should be used where a property is interested in fallback values if actual values are not present.

Cleans up the driver and any fetch rewrite contexts, unless the fetch rewrite got detached by a call to DetachFetch(), in which case a call to DetachedFetchComplete() must also be performed.

void net_instaweb::RewriteDriver::FetchInPlaceResource ( const GoogleUrl &  gurl,
bool  proxy_mode,
AsyncFetch async_fetch 
)

Initiates an In-Place Resource Optimization (IPRO) fetch (A resource which is served under the original URL, but is still able to be rewritten).

proxy_mode indicates whether we are running as a proxy where users depend on us to send contents. When set true, we will perform HTTP fetches to get contents if not in cache and will ignore kRecentFetchNotCacheable and kRecentFetchFailed since we'll have to fetch the resource for users anyway. Origin implementations (like mod_pagespeed) should set this to false and let the serve serve the resource if it's not in cache.

If proxy_mode is false and the resource could not be found in HTTP cache, async_fetch->Done(false) will be called and async_fetch->status_code() will be CacheUrlAsyncFetcher::kNotInCacheStatus (to distinguish this from a different reason for failure, like kRecentFetchNotCacheable).

Note that if the request headers have not yet been set on the driver then they'll be taken from the fetch.

bool net_instaweb::RewriteDriver::FetchOutputResource ( const OutputResourcePtr &  output_resource,
RewriteFilter filter,
AsyncFetch async_fetch 
)

See FetchResource. There are two differences: 1. It takes an OutputResource instead of a URL. 2. It returns whether a fetch was queued or not. This is safe to ignore because in either case the callback will be called. 3. If 'filter' is NULL then the request only checks cache and (if enabled) the file system.

bool net_instaweb::RewriteDriver::FetchResource ( const StringPiece &  url,
AsyncFetch fetch 
)

Initiates an async fetch for a rewritten resource with the specified name. If url matches the pattern of what the driver is authorized to serve, then true is returned and the caller must listen on the callback for the completion of the request.

If the driver is not authorized to serve the resource for any of the following reasons, false is returned and the callback will -not- be called - the request should be passed to another handler. The URL is invalid or it does not match the general pagespeed pattern. The filter id in the URL does not map to a known filter. The filter for the id in the URL doesn't recognize the format of the URL. The filter for the id in the URL is forbidden.

In other words there are three outcomes for this routine: 1. the request was handled immediately and the callback called before the method returns. true is returned. 2. the request looks good but was queued because some other resource fetch is needed to satisfy it. true is returned. 3. the request does not look like it belongs to Instaweb. The callback will not be called, and false will be returned.

In even other words, if this routine returns 'false' then the callback will not be called. If the callback -is- called, then this should be the 'final word' on this request, whether it was called with success=true or success=false.

Note that if the request headers have not yet been set on the driver then they'll be taken from the fetch.

virtual void net_instaweb::RewriteDriver::FinishParse ( ) [virtual]

Override HtmlParse's FinishParse to ensure that the request-scoped cache is cleared immediately.

Note that the RewriteDriver can delete itself in this method, if it's not externally managed, and if all RewriteContexts have been completed.

void net_instaweb::RewriteDriver::FinishParseAsync ( Function *  callback)

As above, but asynchronous. Note that the RewriteDriver may already be deleted at the point the callback is invoked. The scheduler lock will not be held when the callback is run.

Return true if we must flatten css imports, either because the filter is enabled explicitly or because it is enabled by CriticalSelectorsEnabled.

virtual void net_instaweb::RewriteDriver::Flush ( ) [virtual]

Overrides HtmlParse::Flush so that it can happen in two phases: 1. Pre-render chain runs, resulting in async rewrite activity 2. async rewrite activity ends, calling callback, and post-render filters run. This API is used for unit-tests & Apache (which lacks a useful event model) and results in blocking behavior.

FlushAsync is prefered for event-driven servers.

void net_instaweb::RewriteDriver::FlushAsync ( Function *  done)

Initiates an asynchronous Flush. done->Run() will be called when the flush is complete. Further calls to ParseText should be deferred until the callback is called. Scheduler mutex is not held while done is called.

Increment reference count for misc. async ops that need the RewriteDriver kept alive.

We expect to this method to be called on the HTML parser thread or after parsing is completed.

void net_instaweb::RewriteDriver::InfoAt ( const RewriteContext context,
const char *  msg,
  ... 
)

Report error message with description of context's location (such as filenames and line numbers). context may be NULL, in which case the current parse position will be used.

Method to start a resource rewrite. This is called by a filter during parsing, although the Rewrite might continue after deadlines expire and the rewritten HTML must be flushed. Returns InitiateRewrite returns false if the system is not healthy enough to support resource rewrites.

log_record() always returns a pointer to a valid AbstractLogRecord, owned by the rewrite_driver's request context.

bool net_instaweb::RewriteDriver::MatchesBaseUrl ( const GoogleUrl &  input_url) const

Checks to see if the input_url has the same origin as and the base url, to make sure we're not fetching from another server. Does not consult the domain lawyer, and is not affected by AddDomain(). Precondition: input_url.IsWebValid()

Returns true if we may cache extend Css, Images, PDFs, or Scripts respectively.

bool net_instaweb::RewriteDriver::MayRewriteUrl ( const GoogleUrl &  domain_url,
const GoogleUrl &  input_url,
bool  allow_unauthorized_domain,
bool *  is_authorized_domain 
) const

Checks to see if we can write the input_url resource in the domain_url taking into account domain authorization, wildcard allow/disallow from RewriteOptions and allow_unauthorized_domain (a field that is dictated by the filter processing the URL). After the function is executed, is_authorized_domain will indicate whether input_url was found to belong to an authorized domain or not.

bool net_instaweb::RewriteDriver::MetadataRequested ( const RequestHeaders &  request_headers) const

Determines whether metadata was requested in the response headers and verifies that the key in the header is the same as the expected key. An empty expected key returns false.

Determines whether the document's Content-Type has a mimetype indicating that browsers should parse it as XHTML.

Return a mutable pointer to the response headers that filters can update before the first flush. Returns NULL after Flush has occurred.

We expect to this method to be called on the HTML parser thread. Returns the number of pagespeed resources flushed by flush early flow.

We expect to this method to be called on the HTML parser thread. Returns the number of images whose low quality images are inlined in the html page.

void net_instaweb::RewriteDriver::PopulateResourceNamer ( const StringPiece &  filter_id,
const StringPiece &  name,
ResourceNamer full_name 
)

Fills in the resource namer based on the give filter_id, name and options stored in the driver.

Like AppendRewriteFilter, but adds the filter to the beginning of the pre-render chain.

Returns the property page which contains the cached properties associated with the current URL.

RewriteContext* net_instaweb::RewriteDriver::RegisterForPartitionKey ( const GoogleString &  partition_key,
RewriteContext candidate 
)

Tries to register the given rewrite context as working on its partition key. If this context is the first one to try to handle it, returns NULL. Otherwise returns the previous such context.

Must only be called from rewrite thread.

Provides a mechanism for a RewriteContext to notify a RewriteDriver that a certain number of rewrites have been discovered to need to take the slow path.

Indicates that a Flush through the HTML parser chain should happen soon, e.g. once the network pauses its incoming byte stream.

CssResolutionStatus net_instaweb::RewriteDriver::ResolveCssUrls ( const GoogleUrl &  input_css_base,
const StringPiece &  output_css_base,
const StringPiece &  contents,
Writer *  writer,
MessageHandler *  handler 
)

Rewrites CSS content to absolutify any relative embedded URLs, streaming the results to the writer. Returns 'false' if the writer returns false or if the content was not rewritten because the domains of the gurl and resolved_base match.

input_css_base contains the path where the CSS text came from. output_css_base contains the path where the CSS will be written.

const ResponseHeaders* net_instaweb::RewriteDriver::response_headers ( ) [inline]

Returns a const version of the ResponseHeaders*, indepdendent of whether Flush has occurred. Note that ResponseHeaders* may still be NULL if no one has called set_response_headers_ptr.

Todo:
TODO(jmarantz): Change API to require response_headers in StartParse so we can guarantee this is non-null.
void net_instaweb::RewriteDriver::RewriteComplete ( RewriteContext rewrite_context,
bool  permit_render 
)

Provides a mechanism for a RewriteContext to notify a RewriteDriver that it is complete, to allow the RewriteDriver to delete itself or return it back to a free pool in the ServerContext.

This will also call back into RewriteContext::Propagate, letting it know whether the context is still attached to the HTML DOM (and hence safe to render), and to do other bookkeeping.

If 'permit_render' is false, no rendering will be asked for even if the context is still attached.

void net_instaweb::RewriteDriver::SaveOriginalHeaders ( const ResponseHeaders &  response_headers)

Saves the origin headers for a request in flush_early_info so that it can be used in subsequent request.

void net_instaweb::RewriteDriver::set_critical_css_result ( CriticalCssResult *  critical_css_rules)

Sets the Critical CSS rules info in the driver and the ownership of the rules stays with the driver.

Inserts the critical images present on the requested html page. It takes ownership of critical_images_info. This should only be called by the CriticalImagesFinder, normal users should just be using the automatic management of critical_images_info that CriticalImagesFinder provides.

void net_instaweb::RewriteDriver::set_critical_line_info ( CriticalLineInfo *  critical_line_info)

Inserts the critical images present on the requested html page. It takes the ownership of critical_line_info.

This should only be called by the CriticalSelectorFinder.

Todo:
TODO(jud): Remove when the finders reside in RewriteDriver and manage their own state.

Indicate that this RewriteDriver will be explicitly deleted, and thus should not be auto-deleted at the end of the parse. This is primarily for tests.

Todo:
TODO(jmarantz): Consider phasing this out to make tests behave more like servers.

This is relevant only when fully_rewrite_on_flush is true. When this is set to true, Flush of HTML will not wait for async events while it does wait when it is set to false.

void net_instaweb::RewriteDriver::set_flush_early_render_info ( FlushEarlyRenderInfo *  flush_early_render_info)

Takes the ownership of flush_early_render_info. This method is not thread-safe. Call it only from the html parser thread.

If this is set to true, during a Flush of HTML the system will wait for results of all rewrites rather than just waiting for cache lookups and a small deadline. Note, however, that in very rare circumstances some rewrites may still be dropped due to excessive load.

Note: reset every time the driver is recycled.

Sets a maximum amount of time to process a page across all flush windows; i.e., the entire lifecycle of this driver during a given pageload. A negative value indicates no limit. Setting fully_rewrite_on_flush() overrides this.

Takes ownership of 'options'. pool denotes the pool of rewrite drivers that use these options. May be NULL if using custom options.

Sets whether or not there were references to urls before the base tag (if there is a base tag). This variable has document-level scope. It is reset at the beginning of every document by ScanFilter.

void net_instaweb::RewriteDriver::set_response_headers_ptr ( ResponseHeaders *  headers) [inline]

Set the pointer to the response headers that filters can update before the first flush. RewriteDriver does NOT take ownership of this memory.

void net_instaweb::RewriteDriver::SetRequestHeaders ( const RequestHeaders &  headers)

Reinitializes request_headers_ (a scoped ptr) with a copy of the original request headers. Note that the fetches associated with the driver could be using a modified version of the original request headers. There MUST be at most 1 call to this method after a rewrite driver object has been constructed or recycled.

Sets a server context enabling the rewriting of resources. This will replace any previous server context.

Set a fetcher that will be used by RewriteDriver for current request only (that is, until Clear()). RewriteDriver will take ownership of this fetcher, and will keep it around until Clear(), even if further calls to this method are made.

void net_instaweb::RewriteDriver::SetWriter ( Writer *  writer)

Controls how HTML output is written. Be sure to call this last, after all other filters have been established.

Todo:
TODO(jmarantz): fix this in the implementation so that the caller can install filters in any order and the writer will always be last.
bool net_instaweb::RewriteDriver::ShouldAbsolutifyUrl ( const GoogleUrl &  input_base,
const GoogleUrl &  output_base,
bool *  proxy_mode 
) const

Determines if an URL relative to the given input_base needs to be absolutified given that it will end up under output_base:

  • If we are proxying and input_base isn't proxy encoded, then yes.
  • If we aren't proxying and input_base != output_base, then yes.
  • If we aren't proxying and the domain lawyer will shard or rewrite input_base, then yes. If not NULL also set *proxy_mode to whether proxy mode is active or not.
virtual bool net_instaweb::RewriteDriver::StartParseId ( const StringPiece &  url,
const StringPiece &  id,
const ContentType &  content_type 
) [virtual]

Override HtmlParse's StartParseId to propagate any required options. Note that if this (or other variants) returns true you should use FinishParse(), otherwise Cleanup().

Convenience method to return the trace context from the request_context() if both are configured and NULL otherwise.

void net_instaweb::RewriteDriver::TracePrintf ( const char *  fmt,
  ... 
)

Convenience method to issue a trace annotation if tracing is enabled. If tracing is disabled, this function is a no-op.

void net_instaweb::RewriteDriver::UpdatePropertyValueInDomCohort ( AbstractPropertyPage page,
StringPiece  property_name,
StringPiece  property_value 
)

Update the PropertyValue named 'property_name' in dom cohort with the value 'property_value'. It is the responsibility of the client to ensure that property cache and dom cohort are enabled when this function is called. It is a programming error to call this function when property cache or dom cohort is not available, more so since the value payload has to be serialised before calling this function. Hence this function will DFATAL if property cache or dom cohort is not available.

Wait for outstanding Rewrite to complete. Once the rewrites are complete they can be rendered.

Wait for outstanding rewrite to complete, including any background work that may be ongoing even after results were reported.

Note: while this guarantees that the result of the computation is known, the thread that performed it may still be running for a little bit and accessing the driver.

bool net_instaweb::RewriteDriver::Write ( const ResourceVector &  inputs,
const StringPiece &  contents,
const ContentType *  type,
StringPiece  charset,
OutputResource output 
)

Writes the specified contents into the output resource, and marks it as optimized. 'inputs' described the input resources that were used to construct the output, and is used to determine whether the result can be safely cache extended and be marked publicly cacheable. 'content_type' and 'charset' specify the mimetype and encoding of the contents, and will help form the Content-Type header. 'charset' may be empty when not specified.

Note that this does not escape charset.

Callers should take care that dangerous types like 'text/html' do not sneak into content_type.


Member Data Documentation

This string identifies, for the PropertyCache, a group of properties that are computed from the DOM, and thus can, if desired, be rewritten on every HTML request.

Property Names in DomCohort. Tracks the timestamp when we last received a request for this url.

A list of HTTP request headers. These are the headers which should be passed through from the client request into the ResponseHeaders request_headers sent to the rewrite driver. Headers not in this list will be ignored so there is no need to copy them over.

Todo:
TODO(sligocki): Use these in ProxyInterface flow.

The documentation for this class was generated from the following file:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines