|
Page Speed Optimization Libraries
1.13.35.1
|
All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
|
Page Speed Optimization Libraries
1.13.35.1
|
#include "rewrite_context.h"
Classes | |
| struct | CacheLookupResult |
| class | CacheLookupResultCallback |
| Used for LookupMetadataForOutputResource. More... | |
Public Types | |
| typedef std::vector< InputInfo * > | InputInfoStarVector |
Public Member Functions | |
| RewriteContext (RewriteDriver *driver, RewriteContext *parent, ResourceContext *resource_context) | |
| int | num_slots () const |
| ResourceSlotPtr | slot (int index) const |
| int | num_outputs () const |
| OutputResourcePtr | output (int i) const |
| int | num_output_partitions () const |
| const CachedResult * | output_partition (int i) const |
| CachedResult * | mutable_output_partition (int i) |
| bool | chained () const |
| void | AddSlot (const ResourceSlotPtr &slot) |
| void | RemoveLastSlot () |
| void | AddNestedContext (RewriteContext *context) |
| void | CallFetchInputs () |
| void | CallLockFailed () |
| void | CallStartFetchImpl () |
| void | Initiate () |
| bool | Fetch (const OutputResourcePtr &output_resource, AsyncFetch *fetch, MessageHandler *message_handler) |
| bool | slow () const |
| bool | is_metadata_cache_miss () const |
| This particular rewrite was a metadata cache miss. | |
| bool | has_parent () const |
| Returns true if this is a nested rewriter. | |
| bool | IsNestedIn (StringPiece id) const |
| void | CheckNotFrozen () |
| RewriteContext * | parent () |
| Allows a nested rewriter to walk up its parent hierarchy. | |
| const RewriteContext * | parent () const |
| int | num_nested () const |
| Accessors for the nested rewrites. | |
| RewriteContext * | nested (int i) const |
| RewriteDriver * | Driver () const |
| void | set_force_rewrite (bool x) |
| If called with true, forces a rewrite and re-generates the output. | |
| bool | rewrite_uncacheable () const |
| void | set_rewrite_uncacheable (bool rewrite_uncacheable) |
| const ResourceContext * | resource_context () const |
| GoogleString | ToString () const |
| Returns debug information about this RewriteContext. | |
| GoogleString | ToStringWithPrefix (StringPiece prefix) const |
Static Public Member Functions | |
| static void | InitStats (Statistics *stats) |
| Initializes statistics. | |
Static Public Attributes | |
| static const char | kNumRewritesAbandonedForLockContention [] |
| static const char | kNumDeadlineAlarmInvocations [] |
| static const char | kHashMismatchMessage [] |
Protected Types | |
| typedef std::vector< GoogleUrl * > | GoogleUrlStarVector |
Protected Member Functions | |
| ServerContext * | FindServerContext () const |
| const RewriteOptions * | Options () const |
| OutputPartitions * | partitions () |
| void | AddRecheckDependency () |
| virtual bool | OptimizationOnly () const |
| virtual bool | Partition (OutputPartitions *partitions, OutputResourceVector *outputs) |
| virtual void | PartitionAsync (OutputPartitions *partitions, OutputResourceVector *outputs) |
| void | PartitionDone (RewriteResult result) |
| void | CrossThreadPartitionDone (RewriteResult result) |
| virtual void | Rewrite (int partition_index, CachedResult *partition, const OutputResourcePtr &output)=0 |
| void | RewriteDone (RewriteResult result, int partition_index) |
| virtual bool | SendFallbackResponse (StringPiece output_url_base, StringPiece contents, AsyncFetch *async_fetch, MessageHandler *handler) |
| void | StartNestedTasks () |
| virtual void | Harvest () |
| virtual bool | PolicyPermitsRendering () const =0 |
| bool | AreOutputsAllowedByCsp (CspDirective role) const |
| Helper that checks that all output resources are OK with CSP as given role. | |
| virtual void | Render () |
| virtual void | WillNotRender () |
| virtual void | Cancel () |
| virtual const UrlSegmentEncoder * | encoder () const |
| virtual GoogleString | CacheKeySuffix () const |
| virtual GoogleString | UserAgentCacheKey (const ResourceContext *context) const |
| virtual void | EncodeUserAgentIntoResourceContext (ResourceContext *context) |
| virtual const char * | id () const =0 |
| Returns the filter ID. | |
| virtual OutputResourceKind | kind () const =0 |
| void | AttachDependentRequestTrace (const StringPiece &label) |
| RequestTrace * | dependent_request_trace () |
| void | TracePrintf (const char *fmt,...) |
| virtual void | StartFetchReconstruction () |
| void | DetachFetch () |
| virtual bool | DecodeFetchUrls (const OutputResourcePtr &output_resource, MessageHandler *message_handler, GoogleUrlStarVector *url_vector) |
| virtual void | FixFetchFallbackHeaders (const CachedResult &cached_result, ResponseHeaders *headers) |
| virtual void | FetchCallbackDone (bool success) |
| virtual void | FetchTryFallback (const GoogleString &url, const StringPiece &hash) |
| void | Freshen () |
| Freshens resources proactively to avoid expiration in the near future. | |
| bool | notify_driver_on_fetch_done () const |
| void | set_notify_driver_on_fetch_done (bool value) |
| AsyncFetch * | async_fetch () |
| Note that the following must only be called in the fetch flow. | |
| bool | FetchContextDetached () |
| Is fetch_ detached? Only call this in the fetch flow. | |
| MessageHandler * | fetch_message_handler () |
| The message handler for the fetch. | |
| bool | stale_rewrite () const |
| Indicates whether we are serving a stale rewrite. | |
| virtual int64 | GetRewriteDeadlineAlarmMs () const |
| virtual bool | CreationLockBeforeStartFetch () const |
| Should the context call LockForCreation before checking the cache? | |
| virtual bool | FailOnHashMismatch () const |
| virtual bool | ScheduleViaCentralController () |
| virtual bool | ScheduleNestedContextViaCentalController () const |
| void | ObtainLockForCreation (ServerContext *server_context, Function *callback) |
| void | ReleaseCreationLock (bool succeeded) |
Static Protected Member Functions | |
| static bool | LookupMetadataForOutputResourceImpl (OutputResourcePtr output_resource, const GoogleUrl &gurl, RewriteContext *rewrite_context, RewriteDriver *driver, GoogleString *error_out, CacheLookupResultCallback *callback) |
Friends | |
| class | RewriteDriver |
RewriteContext manages asynchronous rewriting of some n >= 1 resources (think CSS, JS, or images) into m >= 0 improved versions (typically, n = m = 1). It also helps update the references in the containing document (called slots), such as <img src=> in HTML, or background-image: url() in CSS, and make any other changes to it needed to commit the optimization.
It is normally used as a base class, with its own code helping take care of caching, fetching, etc., while subclasses describe how to transform the resources, and how to update the document containing them with the new version by overriding some virtuals like Rewrite() and Render().
Filters parsing HTML create their RewriteContext subclasses for every group of resources they think should be optimized together (such as one RewriteContext for every image for image re-compression, or one for a group of CSS files that have compatible HTML markup for CSS combining). The framework may also ask a filter to make its RewriteContext subclass via MakeRewriteContext() in case it need to reconstruct an optimized resource that's not available in the cache.
In the case of combining filters, a single RewriteContext may result in multiple rewritten resources that are partitioned based on data semantics. Most filters will just work on one resource, and those can inherit from SingleRewriteContext which is simpler to implement.
The most basic transformation steps subclasses will want to implement are:
Partition: Determines how many outputs, if any, will be created from all the inputs. For example, a spriter may create separate partitions for groups of images with similar colormaps. This step is also responsible for deciding what to do if some inputs were not loaded successfully. SingleRewriteContext provides the correct implementation for transformations that take in one file and optimize it.
Rewrite: Takes inputs from one partition, and tries to produce an optimized output for it, as well as a CachedResult, which caches any auxiliary information that may be needed to update the container document. For example, the image filter will store image dimensions inside the CachedResult object.
If a better version can be created, the subclass should call RewriteDriver::Write with its data, and then RewriteDone(kRewriteOk).
If no improvement is possible, it should call RewriteDone(kRewriteFailed). Note that this does not mean that nothing can be done, just that no new resource has been created (for example an image filter might still insert dimensions into the tag even if it can't compress the image better).
Render: Updates the document based on information stored in CachedResult. This is the only step that can touch the HTML DOM. Note that you do not need to implement it if you just want to update the URL to the new version: the ResourceSlot's will do it automatically.
Which of the steps get invoked depends on how much information has been cached, as well as on timing of things (since the system tries not to hold up the web page noticeably to wait for an optimization). Common scenarios are:
1) New rewrite, finishes quickly: Partition -> Rewrite -> Render 2) New rewrite, but too slow to render: Partition -> Rewrite 3) Metadata cache hit: Render 4) Reconstructing output from a .pagespeed. URL: Rewrite
Note in particular that (3) means that all rendering should be doable just from information inside the CachedResult.
Top-level RewriteContexts are initialized from the HTML thread, by filters responding to parser events. In particular, from this thread they can be constructed, and AddSlot() and Initiate() can be called. Once Initiate is called, the RewriteContext runs purely in its two threads, until it completes. At that time it will self-delete in coordination with RewriteDriver.
RewriteContexts can also be nested, in which case they are constructed, slotted, and Initated all within the rewrite threads. However, they are Propagated and destructed by their parent, which was initiated by the RewriteDriver.
RewriteContext utilizes two threads (via QueuedWorkerPool::Sequence) to do most of its work. The "high priority" thread is used to run the dataflow graph: queue up fetches and cache requests, partition inputs, render results, etc. The actual Rewrite() methods, however, are invoked in the "low priority" thread and can be canceled during extreme load or shutdown.
| net_instaweb::RewriteContext::RewriteContext | ( | RewriteDriver * | driver, |
| RewriteContext * | parent, | ||
| ResourceContext * | resource_context | ||
| ) |
Takes ownership of resource_context, which must be NULL or allocated with 'new'.
| driver | exactly one of driver & parent |
| parent | is non-null |
| void net_instaweb::RewriteContext::AddNestedContext | ( | RewriteContext * | context | ) |
Adds a new nested RewriteContext. This RewriteContext will not be considered complete until all nested contexts have completed. This may be useful, for example for a CSS optimizer that also wants to optimize images referred to from CSS (in which case the image rewrite context will be nested inside the CSS context).
|
protected |
Add a dummy other_dependency that will force the rewrite's OutputPartitions to be rechecked after a modest TTL.
| void net_instaweb::RewriteContext::AddSlot | ( | const ResourceSlotPtr & | slot | ) |
Resource slots must be added to a Rewrite before Initiate() can be called. Starting the rewrite sets in motion a sequence of async cache-lookups &/or fetches.
|
protected |
Tracing API.Creates a new request trace associated with this context with a given |label|.
|
protectedvirtual |
Allows subclasses to add additional text to be appended to the metadata cache key. The default implementation returns "".
Reimplemented in net_instaweb::CssFilter::Context, and net_instaweb::CssFlattenImportsContext.
|
protectedvirtual |
This method is invoked (in Rewrite thread) if this context got canceled due to an earlier filter sharing a slot with it having called set_disable_further_processing. Default implementation does nothing.
Reimplemented in net_instaweb::CombiningFilter::Context.
|
inline |
Returns true if this context is chained to some predecessors, and must therefore be started by a predecessor and not RewriteDriver.
| void net_instaweb::RewriteContext::CheckNotFrozen | ( | ) |
Checks to make sure that partitions_ is not frozen when it is about to be modified, calling LOG(DFATAL) if there is a problem.
|
protected |
Helper for queuing invocation of PartitionDone to run in the main rewrite sequence.
|
protectedvirtual |
Decodes the output resource to find the resources to be fetched. The default behavior decodes the output resource name into multiple paths and absolutifies them with respect to the output resource base. Returns true if the decoding is successful and false otherwise.
Reimplemented in net_instaweb::InPlaceRewriteContext.
|
inlineprotected |
Provides the dependent request trace associated with this context, if any. Note that this is distinct from the root user request trace, available in Driver().
|
protected |
Makes the rest of a fetch run in background, not producing a result or invoking callbacks. Will arrange for appropriate memory management with the rewrite driver itself; but the caller is responsible for delivering results itself and invoking the callback.
|
protectedvirtual |
This final set of protected methods can be optionally overridden by subclasses. All RewriteContexts define how they encode URLs and other associated information needed for a rewrite into a URL. The default implementation handles a single URL with no extra data. The RewriteContext owns the encoder.
Reimplemented in net_instaweb::CombiningFilter::Context, and net_instaweb::CssFilter::Context.
|
inlineprotectedvirtual |
Encodes User Agent into the ResourceContext. A subclass ResourceContext should normally call RewriteFilter::EncodeUserAgentIntoResourceContext if it has access to a RewriteFilter.
Reimplemented in net_instaweb::InPlaceRewriteContext.
|
inlineprotectedvirtual |
Should the context fail to serve the rewritten resource if the hash doesn't match user requested hash? By default, we do not fail and simply serve with limited Caching headers assuming that an out-of-date resource is better than none. But for resources like source maps, out-of-date versions are worse than nothing because they are complete non-sense if not associated with the exact expected contents.
| bool net_instaweb::RewriteContext::Fetch | ( | const OutputResourcePtr & | output_resource, |
| AsyncFetch * | fetch, | ||
| MessageHandler * | message_handler | ||
| ) |
Fetch the specified output resource by reconstructing it from its inputs, sending output into fetch.
True is returned if an asynchronous fetch got queued up. If false, fetch->Done() will not be called.
|
protectedvirtual |
Callback once the fetch is done. This calls Driver()->FetchComplete() if notify_driver_on_fetch_done is true.
|
protectedvirtual |
Attempts to fetch a given URL from HTTP cache, and serves it (with shortened HTTP headers) if available. If not, fallback to normal full reconstruction path. Note that the hash can be an empty string if the url is not rewritten.
|
protected |
Resource transformation APIs. If you are implementing an optimization, you'll be dealing mainly with these.Finds the ServerContext associated with this context. Note that this method might have to climb up the parent-tree, but it's typically not a deep tree. Same with Driver() and Options().
|
protectedvirtual |
Adjust headers sent out for a stale or in-place result. We may send out stale results in the fallback fetch pathway, but these results should not be cached much. By default we strip Set-Cookie* headers and Etags, and convert Cache-Control headers to private, max-age=300.
|
protectedvirtual |
Returns an interval in milliseconds to wait when configuring the deadline alarm in FetchContext::SetupDeadlineAlarm(). Subclasses may configure the deadline based on rewrite type, e.g., IPRO vs. HTML-path.
Reimplemented in net_instaweb::InPlaceRewriteContext.
|
protectedvirtual |
Once any nested rewrites have completed, the results of these can be incorporated into the rewritten data. For contexts that do not require any nested RewriteContexts, it is OK to skip overriding this method – the empty default implementation is fine.
Reimplemented in net_instaweb::CssFilter::Context, net_instaweb::NestedFilter::Context, and net_instaweb::CssFlattenImportsContext.
| void net_instaweb::RewriteContext::Initiate | ( | ) |
Starts a resource rewrite. Once Inititated, the Rewrite object should only be accessed from the Rewrite thread, until it Completes, at which point top-level Contexts will call RewriteComplete on their driver, and nested Contexts will call NestedRewriteComplete on their parent. Nested rewrites will be Started directly from their parent context, and Initiate will not be called.
Precondition: this rewrite isn't anyone's successor (e.g. chain() == false) and has not been started before.
| bool net_instaweb::RewriteContext::IsNestedIn | ( | StringPiece | id | ) | const |
Returns true if this is a child rewriter and its parent has the given id.
|
protectedpure virtual |
Rewrites come in three flavors, as described in output_resource_kind.h, so this method must be defined by subclasses to indicate which it is.
For example, we will avoid caching output_resource content in the HTTP cache for rewrites that are so quick to complete that it's fine to do the rewrite on every request. extend_cache is obviously in this category, and it's arguable we could treat js minification that way too (though we don't at the moment).
Implemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::NestedFilter::Context, net_instaweb::CssFlattenImportsContext, net_instaweb::InPlaceRewriteContext, net_instaweb::SimpleTextFilter::Context, net_instaweb::InlineRewriteContext, and net_instaweb::FakeFilter::Context.
|
staticprotected |
Backend to RewriteDriver::LookupMetadataForOutputResource, with the RewriteContext of appropriate type and the OutputResource already created. Takes ownership of rewrite_context.
| int net_instaweb::RewriteContext::num_output_partitions | ( | ) | const |
These are generally accessed in the Rewrite thread, but may also be accessed in ::Render.
|
inline |
Random access to outputs. These should only be accessed by the RewriteThread.
|
inline |
Random access to slots. This is not thread-safe. Prior to Initialize(), these can be called by the constructing thread. After Initiate(), these should only be called by the Rewrite thread.
|
protected |
Obtain a lock to create the resource. callback may not be invoked for an indeterminate time.
|
inlineprotectedvirtual |
If this returns true, running the rewriter isn't required for correctness of the page, so the engine will be permitted to drop the rewrite if needed to preserve system responsiveness.
Reimplemented in net_instaweb::CombiningFilter::Context, and net_instaweb::SimpleTextFilter::Context.
|
protectedvirtual |
Partitions the input resources into one or more outputs. Return 'true' if the partitioning could complete (whether a rewrite was found or not), false if the attempt was abandoned and no conclusion can be drawn.
Note that if partitioner finds that the resources are not rewritable, it will still return true; it will simply have an empty inputs-array in OutputPartitions and leave 'outputs' unmodified. 'false' is only returned if the subclass skipped the rewrite attempt due to a lock conflict.
You must override one of Partition() or PartitionAsync(). Partition() is normally fine unless you need to do computations that can take a noticeable amount of time, since there are some scenarios under which page output may end up being held up for a partitioning step. If you do need to do something computationally expensive in partitioning steps, override PartitionAsync() instead.
Reimplemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::InlineRewriteContext, and net_instaweb::SingleRewriteContext.
|
protectedvirtual |
As above, but you report the result asynchronously by calling PartitionDone(), which must be done from the main rewrite sequence. One of Partition or PartitionAsync() must be overridden in the subclass. The default implementation is implemented in terms of Partition().
|
protected |
Call this from the main rewrite sequence to report results of PartitionAsync. If the client is not in the main rewrite sequence, use CrossThreadPartitionDone() instead.
|
protectedpure virtual |
This method gives the context a chance to verify that rendering the result is consistent with the current document's (Content Security) Policy, which may be different than that of the page for which the result was first computed + cached. Most subclasses can just call AreOutputsAllowedByCsp(), with appropriate role.
Implemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::NestedFilter::Context, net_instaweb::CssFlattenImportsContext, net_instaweb::SimpleTextFilter::Context, and net_instaweb::FakeFilter::Context.
|
protected |
Release whichever lock was obtained above. succeeded will be used to inform the CentralController if it should retry (when success = false). If this is not explicitly called, the lock will be released when "this" is destroyed.
| void net_instaweb::RewriteContext::RemoveLastSlot | ( | ) |
Remove the last slot from the context's slot list. This context must be the last one attached to the slot.
|
protectedvirtual |
Performs rendering activities that span multiple HTML slots. For example, in a filter that combines N slots to 1, N-1 of the HTML elements might need to be removed. That can be performed in Render(). This method is optional; the base-class implementation is empty.
Note that unlike Harvest(), this method runs in the HTML thread (for top-level rewrites), and only runs if the rewrite completes prior to the rewrite-deadline. If the rewrite does make it by the deadline, RewriteContext::Render() will be invoked regardless of whether any slots were actually optimized successfully.
Reimplemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::CssFlattenImportsContext, and net_instaweb::InlineRewriteContext.
|
protectedpure virtual |
Takes a completed rewrite partition and rewrites it. When complete, implementations should call RewriteDone(kRewriteOk) if they successfully created an output resource using RewriteDriver::Write, and RewriteDone(kRewriteFailed) if they didn't. They may also call RewriteDone(kTooBusy) in case system load/resource usage makes it dangerous for the filter to do optimization at this time.
Any information about the inputs or output that may be needed to update the containing document should be stored inside the CachedResult.
If implementors wish to rewrite resources referred to from within the inputs (e.g. images in CSS), they may create nested rewrite contexts and call AddNestedContext() on each, and then StartNestedTasks() when all have been added.
Implemented in net_instaweb::CombiningFilter::Context, net_instaweb::InlineRewriteContext, and net_instaweb::SingleRewriteContext.
|
protected |
Called by subclasses when an individual rewrite partition is done. Note that RewriteDone may 'delete this' so no further references to 'this' should follow a call to RewriteDone. This method can run in any thread.
|
inlineprotectedvirtual |
In general, ScheduleViaCentralController() is ignored for nested Contexts. However, in the case of (at least) IPRO we need to schedule the inner context via the Controller. This can be overridden by such contexts, which are DHCHECKed to have at most one nested context. See longer comment in ObtainLockForCreation implementation.
Reimplemented in net_instaweb::InPlaceRewriteContext.
|
inlineprotectedvirtual |
Whether the CentralController should be used to schedule this rewrite. Expensive RewriteContexts (CSS, Images) should override this to return true, allowing more intelligent prioritization.
|
protectedvirtual |
Sends a a response to the the client via the AsyncFetch, transforming output if needed (e.g. css absolutification) and controlling chunked encoding hints as needed.
This is called in case a rewrite fails in the fetch path or a deadline is exceeded. Default implementation is just to write the input. But contexts may need to specialize this to actually absolutify subresources if the fetched resource is served on a different path than the input resource.
Reimplemented in net_instaweb::CssFilter::Context.
|
inline |
If true, we have determined that this job can't be rendered just from metadata cache (including all prerequisites).
|
protectedvirtual |
Fetch state machine override APIs, as well as exports of some general state machine state for overriders to use. If you just want to write an optimization, you do not need these — they are useful if you want to write a new state machine that's similar but not quite identical to what RewriteContext provides.Called in fetch path if we have not found the resource available in HTTP cache under an alternate location suggested by metadata cache such as a different hash or the original, and thus need to fully reconstruct it.
The base implementation will do an asynchronous locking attempt, scheduling to run FetchInputs when complete. Subclasses may override this method to preload inputs in a different manner, and may delay calling of base version until that is complete.
Reimplemented in net_instaweb::InPlaceRewriteContext.
|
protected |
Called on the parent to initiate all nested tasks. This is so that they can all be added before any of them are started. May be called from any thread.
|
protected |
A convenience wrapper to log a trace annotation in both the request trace (if present) as well as the root user request trace (if present).
|
inlineprotectedvirtual |
Indicates user agent capabilities that must be stored in the cache key.
Note that the context may be NULL as it may not be set before this. Since it isn't going to be modified in the method, ResourceContext is passed as a const pointer.
Reimplemented in net_instaweb::CssFilter::Context, net_instaweb::InPlaceRewriteContext, and net_instaweb::FakeFilter::Context.
|
protectedvirtual |
Notifies the subclass that the filter will not be able to render its output to the containing HTML document, because it wasn't ready in time. Note that neither Render() nor WillNotRender() may be called in case this rewrite got canceled due to disable_further_processing(), or in case Partition() failed. This is called from the HTML thread, but should only be used for read access, and subclasss implementations are required to be reasonably quick since it's called with rewrite_mutex() held. It's called after any earlier contexts in filter order had completed their rendering, if any, but with no order guarantees with respect to other WillNotRender() invocations.
Reimplemented in net_instaweb::CombiningFilter::Context.
1.8.6