Page Speed Optimization Libraries  1.4.26.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines
Classes | Public Types | Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Types | Protected Member Functions | Friends
net_instaweb::RewriteContext Class Reference

#include "rewrite_context.h"

Inheritance diagram for net_instaweb::RewriteContext:
net_instaweb::CombiningFilter::Context net_instaweb::InlineRewriteContext net_instaweb::SingleRewriteContext net_instaweb::CssFilter::Context net_instaweb::CssFlattenImportsContext net_instaweb::InPlaceRewriteContext net_instaweb::NestedFilter::Context net_instaweb::SimpleTextFilter::Context

List of all members.

Classes

struct  CacheLookupResult
class  CacheLookupResultCallback
 Used for LookupMetadataForOutputResource. More...

Public Types

typedef std::vector< InputInfo * > InputInfoStarVector

Public Member Functions

 RewriteContext (RewriteDriver *driver, RewriteContext *parent, ResourceContext *resource_context)
int num_slots () const
ResourceSlotPtr slot (int index) const
int num_outputs () const
OutputResourcePtr output (int i) const
int num_output_partitions () const
const CachedResult * output_partition (int i) const
CachedResult * output_partition (int i)
bool chained () const
void AddSlot (const ResourceSlotPtr &slot)
void RemoveLastSlot ()
void Initiate ()
bool Fetch (const OutputResourcePtr &output_resource, AsyncFetch *fetch, MessageHandler *message_handler)
void Propagate (bool render_slots)
bool slow () const
bool has_parent () const
 Returns true if this is a nested rewriter.
RewriteContextparent ()
 Allows a nested rewriter to walk up its parent hierarchy.
const RewriteContextparent () const
void AddNestedContext (RewriteContext *context)
void set_force_rewrite (bool x)
 If called with true, forces a rewrite and re-generates the output.
const ResourceContext * resource_context () const
bool is_metadata_cache_miss () const
bool rewrite_uncacheable () const
void set_rewrite_uncacheable (bool rewrite_uncacheable)
void DetachSlots ()
GoogleString ToString (StringPiece prefix) const
 Returns debug information about this RewriteContext.

Static Public Member Functions

static bool LookupMetadataForOutputResource (const GoogleString &url, RewriteDriver *driver, GoogleString *error_out, CacheLookupResultCallback *callback)
static void InitStats (Statistics *stats)
 Initializes statistics.

Static Public Attributes

static const char kNumDeadlineAlarmInvocations []

Protected Types

typedef std::vector< GoogleUrl * > GoogleUrlStarVector

Protected Member Functions

void AttachDependentRequestTrace (const StringPiece &label)
RequestTracedependent_request_trace ()
void TracePrintf (const char *fmt,...)
ServerContextFindServerContext () const
 The following methods are provided for the benefit of subclasses.
const RewriteOptionsOptions () const
RewriteDriverDriver () const
void AddRecheckDependency ()
void RenderPartitionOnDetach (int partition_index)
void RewriteDone (RewriteResult result, int partition_index)
void NestedRewriteDone (const RewriteContext *context)
void StartNestedTasks ()
bool CreateOutputResourceForCachedOutput (const CachedResult *cached_result, bool force_hash_to_zero, OutputResourcePtr *output_resource)
virtual bool OptimizationOnly () const
virtual bool Partition (OutputPartitions *partitions, OutputResourceVector *outputs)
virtual void PartitionAsync (OutputPartitions *partitions, OutputResourceVector *outputs)
void PartitionDone (bool result)
void CrossThreadPartitionDone (bool result)
virtual void Rewrite (int partition_index, CachedResult *partition, const OutputResourcePtr &output)=0
virtual void Harvest ()
virtual void Render ()
virtual const UrlSegmentEncoderencoder () const
virtual GoogleString CacheKeySuffix () const
virtual const char * id () const =0
 Returns the filter ID.
virtual OutputResourceKind kind () const =0
virtual void StartFetchReconstruction ()
void DetachFetch ()
virtual bool DecodeFetchUrls (const OutputResourcePtr &output_resource, MessageHandler *message_handler, GoogleUrlStarVector *url_vector)
virtual void FixFetchFallbackHeaders (ResponseHeaders *headers)
virtual void FetchCallbackDone (bool success)
virtual bool AbsolutifyIfNeeded (const StringPiece &input_contents, Writer *writer, MessageHandler *handler)
virtual void FetchTryFallback (const GoogleString &url, const StringPiece &hash)
void Freshen ()
 Freshens resources proactively to avoid expiration in the near future.
int num_nested () const
 Accessors for the nested rewrites.
RewriteContextnested (int i) const
OutputPartitions * partitions ()
void set_notify_driver_on_fetch_done (bool value)
AsyncFetchasync_fetch ()
 Note that the following must only be called in the fetch flow.
bool FetchContextDetached ()
 Is fetch_ detached? Only call this in the fetch flow.
MessageHandlerfetch_message_handler ()
 The message handler for the fetch.
bool stale_rewrite () const
 Indicates whether we are serving a stale rewrite.
virtual int64 GetRewriteDeadlineAlarmMs () const
virtual GoogleString UserAgentCacheKey (const ResourceContext *context) const
virtual void EncodeUserAgentIntoResourceContext (ResourceContext *context)

Friends

class OutputCacheCallback
class HTTPCacheCallback
class ResourceCallbackUtils
class ResourceRevalidateCallback
class InvokeRewriteFunction

Detailed Description

A RewriteContext is all the contextual information required to perform one or more Rewrites. Member data in the ResourceContext helps us track the collection of data to rewrite, via async cache-lookup or async fetching. It also tracks what to do with the rewritten data when the rewrite completes (e.g. rewrite the URL in HTML or serve the requested data).

RewriteContext is subclassed to control the transformation (e.g. minify js, compress images, etc).

A new RewriteContext is created on behalf of an HTML or CSS rewrite, or on behalf of a resource-fetch. A single filter may have multiple outstanding RewriteContexts associated with it. In the case of combining filters, a single RewriteContext may result in multiple rewritten resources that are partitioned based on data semantics. Most filters will just work on one resource, and those can inherit from SingleRewriteContext which is simpler to implement.

Todo:
TODO(jmarantz): add support for controlling TTL on failures.

RewriteContext utilizes two threads (via QueuedWorkerPool::Sequence) to do most of its work. The "high priority" thread is used to run the dataflow graph: queue up fetches and cache requests, partition inputs, render results, etc. The actual Rewrite() methods, however, are invoked in the "low priority" thread and can be canceled during extreme load or shutdown.

Top-level RewriteContexts are initialized from the HTML thread. In particular, from this thread they can be constructed, and AddSlot() and Initiate() can be called. Once Initiate is called, the RewriteContext runs purely in its two threads, until it completes. At that time it calls RewriteDriver::RewriteComplete. Once complete, the RewriteDriver can call RewriteContext::Propagate() and finally delete the object.

RewriteContexts can also be nested, in which case they are constructed, slotted, and Initated all within the rewrite threads. However, they are Propagated and destructed by their parent, which is initiated by the RewriteDriver.


Constructor & Destructor Documentation

net_instaweb::RewriteContext::RewriteContext ( RewriteDriver driver,
RewriteContext parent,
ResourceContext *  resource_context 
)

Takes ownership of resource_context, which must be NULL or allocated with 'new'.

Parameters:
driverexactly one of driver & parent
parentis non-null

Member Function Documentation

virtual bool net_instaweb::RewriteContext::AbsolutifyIfNeeded ( const StringPiece &  input_contents,
Writer writer,
MessageHandler handler 
) [protected, virtual]

Absolutify contents of an input resource and write it into writer. This is called in case a rewrite fails in the fetch path or a deadline is exceeded. Default implementation is just to write the input. But contexts may need to specialize this to actually absolutify subresources if the fetched resource is served on a different path than the input resource.

Reimplemented in net_instaweb::CssFilter::Context.

Adds a new nested RewriteContext. This RewriteContext will not be considered complete until all nested contexts have completed.

Add a dummy other_dependency that will force the rewrite's OutputPartitions to be rechecked after a modest TTL.

Resource slots must be added to a Rewrite before Initiate() can be called. Starting the rewrite sets in motion a sequence of async cache-lookups &/or fetches.

void net_instaweb::RewriteContext::AttachDependentRequestTrace ( const StringPiece &  label) [protected]

Creates a new request trace associated with this context with a given |label|.

virtual GoogleString net_instaweb::RewriteContext::CacheKeySuffix ( ) const [protected, virtual]

Allows subclasses to add additional text to be appended to the metadata cache key. The default implementation returns "".

Reimplemented in net_instaweb::CssFilter::Context, and net_instaweb::CssFlattenImportsContext.

bool net_instaweb::RewriteContext::chained ( ) const [inline]

Returns true if this context is chained to some predecessors, and must therefore be started by a predecessor and not RewriteDriver.

bool net_instaweb::RewriteContext::CreateOutputResourceForCachedOutput ( const CachedResult *  cached_result,
bool  force_hash_to_zero,
OutputResourcePtr output_resource 
) [protected]

Deconstructs a URL by name and creates an output resource that corresponds to it. If force_hash_to_zero is present, we are processing a stale rewrite, and we should use "0" as the hash so that the client browser sends a request for the freshened rewritten resource.

void net_instaweb::RewriteContext::CrossThreadPartitionDone ( bool  result) [protected]

Helper for queuing invocation of PartitionDone to run in the main rewrite sequence.

virtual bool net_instaweb::RewriteContext::DecodeFetchUrls ( const OutputResourcePtr output_resource,
MessageHandler message_handler,
GoogleUrlStarVector *  url_vector 
) [protected, virtual]

Decodes the output resource to find the resources to be fetched. The default behavior decodes the output resource name into multiple paths and absolutifies them with respect to the output resource base. Returns true if the decoding is successful and false otherwise.

Reimplemented in net_instaweb::InPlaceRewriteContext.

Provides the dependent request trace associated with this context, if any. Note that this is distinct from the root user request trace, available in Driver().

Makes the rest of a fetch run in background, not producing a result or invoking callbacks. Will arrange for appropriate memory management with the rewrite driver itself; but the caller is responsible for delivering results itself and invoking the callback.

Removes this RewriteContext from all slots. This is done normally when a RewriteContext is completed and we are ready to run the successors. It is also done when aborting a RewriteContext due to cache being unhealthy.

virtual const UrlSegmentEncoder* net_instaweb::RewriteContext::encoder ( ) const [protected, virtual]

This final set of protected methods can be optionally overridden by subclasses. All RewriteContexts define how they encode URLs and other associated information needed for a rewrite into a URL. The default implementation handles a single URL with no extra data. The RewriteContext owns the encoder.

Todo:
TODO(jmarantz): remove the encoder from RewriteFilter.

Reimplemented in net_instaweb::CombiningFilter::Context, and net_instaweb::CssFilter::Context.

virtual void net_instaweb::RewriteContext::EncodeUserAgentIntoResourceContext ( ResourceContext *  context) [inline, protected, virtual]

Encodes User Agent into the ResourceContext. A subclass ResourceContext should normally call RewriteFilter::EncodeUserAgentIntoResourceContext if it has access to a RewriteFilter.

Reimplemented in net_instaweb::InPlaceRewriteContext.

bool net_instaweb::RewriteContext::Fetch ( const OutputResourcePtr output_resource,
AsyncFetch fetch,
MessageHandler message_handler 
)

Fetch the specified output resource by reconstructing it from its inputs, sending output into response_writer, writing headers to response_headers, and calling callback->Done(bool success) when complete.

True is returned if an asynchronous fetch got queued up. If false, Done() will not be called.

virtual void net_instaweb::RewriteContext::FetchCallbackDone ( bool  success) [protected, virtual]

Callback once the fetch is done. This calls Driver()->FetchComplete() if notify_driver_on_fetch_done is true.

virtual void net_instaweb::RewriteContext::FetchTryFallback ( const GoogleString url,
const StringPiece &  hash 
) [protected, virtual]

Attempts to fetch a given URL from HTTP cache, and serves it (with shortened HTTP headers) if available. If not, fallback to normal full reconstruction path. Note that the hash can be an empty string if the url is not rewritten.

The following methods are provided for the benefit of subclasses.

Finds the ServerContext associated with this context. Note that this method might have to climb up the parent-tree, but it's typically not a deep tree. Same with Driver() and Options().

virtual void net_instaweb::RewriteContext::FixFetchFallbackHeaders ( ResponseHeaders headers) [protected, virtual]

Fixes the headers resulting from a fetch fallback. This is called when a fetch fallback is found in cache. The default implementation strips cookies and sets the cache ttl to the implicit cache ttl ms.

virtual int64 net_instaweb::RewriteContext::GetRewriteDeadlineAlarmMs ( ) const [protected, virtual]

Returns an interval in milliseconds to wait when configuring the deadline alarm in FetchContext::SetupDeadlineAlarm(). Subclasses may configure the deadline based on rewrite type, e.g., IPRO vs. HTML-path.

Reimplemented in net_instaweb::InPlaceRewriteContext.

virtual void net_instaweb::RewriteContext::Harvest ( ) [protected, virtual]

Once any nested rewrites have completed, the results of these can be incorporated into the rewritten data. For contexts that do not require any nested RewriteContexts, it is OK to skip overriding this method -- the empty default implementation is fine.

Reimplemented in net_instaweb::CssFilter::Context, net_instaweb::NestedFilter::Context, and net_instaweb::CssFlattenImportsContext.

Starts a resource rewrite. Once Inititated, the Rewrite object should only be accessed from the Rewrite thread, until it Completes, at which point top-level Contexts will call RewriteComplete on their driver, and nested Contexts will call NestedRewriteComplete on their parent. Nested rewrites will be Started directly from their parent context, and Initiate will not be called.

Precondition: this rewrite isn't anyone's successor (e.g. chain() == false) and has not been started before.

virtual OutputResourceKind net_instaweb::RewriteContext::kind ( ) const [protected, pure virtual]

Rewrites come in three flavors, as described in output_resource_kind.h, so this method must be defined by subclasses to indicate which it is.

For example, we will avoid caching output_resource content in the HTTP cache for rewrites that are so quick to complete that it's fine to do the rewrite on every request. extend_cache is obviously in this category, and it's arguable we could treat js minification that way too (though we don't at the moment).

Implemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::NestedFilter::Context, net_instaweb::CssFlattenImportsContext, net_instaweb::InPlaceRewriteContext, net_instaweb::SimpleTextFilter::Context, and net_instaweb::InlineRewriteContext.

static bool net_instaweb::RewriteContext::LookupMetadataForOutputResource ( const GoogleString url,
RewriteDriver driver,
GoogleString error_out,
CacheLookupResultCallback callback 
) [static]

Attempts to lookup the metadata cache info that would be used for the output resource at url with the RewriteOptions set on driver.

If there is a problem with the URL, returns false, and *error_out will contain an error message.

If it can determine the metadata cache key successfully, returns true, and eventually callback will be invoked with the metadata cache key and the decoding results.

Do not use the driver passed to this method for anything else.

Note: this method is meant for debugging use only.

void net_instaweb::RewriteContext::NestedRewriteDone ( const RewriteContext context) [protected]

Called on the parent from a nested Rewrite when it is complete. Note that we don't track rewrite success/failure here. We only care whether the nested rewrites are complete, and whether there are any dependencies.

These are generally accessed in the Rewrite thread, but may also be accessed in ::Render.

Random access to outputs. These should only be accessed by the RewriteThread.

Random access to slots. This is not thread-safe. Prior to Initialize(), these can be called by the constructing thread. After Initiate(), these should only be called by the Rewrite thread.

virtual bool net_instaweb::RewriteContext::OptimizationOnly ( ) const [inline, protected, virtual]

If this returns true, running the rewriter isn't required for correctness of the page, so the engine will be permitted to drop the rewrite if needed to preserve system responsiveness.

Reimplemented in net_instaweb::CombiningFilter::Context, and net_instaweb::SimpleTextFilter::Context.

virtual bool net_instaweb::RewriteContext::Partition ( OutputPartitions *  partitions,
OutputResourceVector *  outputs 
) [protected, virtual]

Partitions the input resources into one or more outputs. Return 'true' if the partitioning could complete (whether a rewrite was found or not), false if the attempt was abandoned and no conclusion can be drawn.

Note that if partitioner finds that the resources are not rewritable, it will still return true; it will simply have an empty inputs-array in OutputPartitions and leave 'outputs' unmodified. 'false' is only returned if the subclass skipped the rewrite attempt due to a lock conflict.

You must override one of Partition() or PartitionAsync(). Partition() is normally fine unless you need to do computations that can take a noticeable amount of time, since there are some scenarios under which page output may end up being held up for a partitioning step. If you do need to do something computationally expensive in partitioning steps, override PartitionAsync() instead.

Reimplemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::InlineRewriteContext, and net_instaweb::SingleRewriteContext.

virtual void net_instaweb::RewriteContext::PartitionAsync ( OutputPartitions *  partitions,
OutputResourceVector *  outputs 
) [protected, virtual]

As above, but you report the result asynchronously by calling PartitionDone(), which must be done from the main rewrite sequence. One of Partition or PartitionAsync() must be overridden in the subclass. The default implementation is implemented in terms of Partition().

void net_instaweb::RewriteContext::PartitionDone ( bool  result) [protected]

Call this from the main rewrite sequence to report results of PartitionAsync. If the client is not in the main rewrite sequence, use CrossThreadPartitionDone() instead.

void net_instaweb::RewriteContext::Propagate ( bool  render_slots)

Runs after all Rewrites have been completed, and all nested RewriteContexts have completed and harvested.

For top-level Rewrites, this must be called from the HTML thread. For nested Rewrites it runs from the Rewrite thread.

If render_slots is true, then all the slots owned by this context will have Render() called on them. For top-level Rewrites, this should only be done if the rewrite completes before the rewrite deadline expires. After that, the HTML elements referred to by the slots have already been flushed to the network. For nested Rewrites it's done unconditionally.

Rewriting and propagation continue even after this deadline, so that we may cache the rewritten results, allowing the deadline to be easier-to-hit next time the same resources need to be rewritten.

And in all cases, the successors Rewrites are queued up in the Rewrite thread once any nested propagation is complete. And, in particular, each slot must be updated with any rewritten resources, before the successors can be run, independent of whether the slots can be rendered into HTML.

Remove the last slot from the context's slot list. This context must be the last one attached to the slot.

virtual void net_instaweb::RewriteContext::Render ( ) [protected, virtual]

Performs rendering activities that span multiple HTML slots. For example, in a filter that combines N slots to 1, N-1 of the HTML elements might need to be removed. That can be performed in Render(). This method is optional; the base-class implementation is empty.

Note that unlike Harvest(), this method runs in the HTML thread (for top-level rewrites), and only runs if the rewrite completes prior to the rewrite-deadline. If the rewrite does make it by the deadline, RewriteContext::Render() will be invoked regardless of whether any slots were actually optimized successfully.

Reimplemented in net_instaweb::CombiningFilter::Context, net_instaweb::CssFilter::Context, net_instaweb::CssFlattenImportsContext, and net_instaweb::InlineRewriteContext.

void net_instaweb::RewriteContext::RenderPartitionOnDetach ( int  partition_index) [protected]

Establishes that a slot has been rewritten. So when Propagate() is called, the resource update that has been written to this slot can be propagated to the DOM.

virtual void net_instaweb::RewriteContext::Rewrite ( int  partition_index,
CachedResult *  partition,
const OutputResourcePtr output 
) [protected, pure virtual]

Takes a completed rewrite partition and rewrites it. When complete calls RewriteDone with kRewriteOk if successful. Note that a value of kTooBusy means that an HTML rewrite will skip this resource, but we should not cache it as "do not optimize".

During this phase, any nested contexts that are needed to complete the Rewrite process can be instantiated.

Todo:
TODO(jmarantz): check for resource completion from a different thread (while we were waiting for resource fetches) when Rewrite gets called.

Implemented in net_instaweb::CombiningFilter::Context, net_instaweb::InlineRewriteContext, and net_instaweb::SingleRewriteContext.

void net_instaweb::RewriteContext::RewriteDone ( RewriteResult  result,
int  partition_index 
) [protected]

Called by subclasses when an individual rewrite partition is done. Note that RewriteDone may 'delete this' so no further references to 'this' should follow a call to RewriteDone. This method can run in any thread.

bool net_instaweb::RewriteContext::slow ( ) const [inline]

If true, we have determined that this job can't be rendered just from metadata cache (including all prerequisites).

virtual void net_instaweb::RewriteContext::StartFetchReconstruction ( ) [protected, virtual]

Called in fetch path if we have not found the resource available in HTTP cache under an alternate location suggested by metadata cache such as a different hash or the original, and thus need to fully reconstruct it.

The base implementation will do an asynchronous locking attempt, scheduling to run FetchInputs when complete. Subclasses may override this method to preload inputs in a different manner, and may delay calling of base version until that is complete.

Reimplemented in net_instaweb::InPlaceRewriteContext.

Called on the parent to initiate all nested tasks. This is so that they can all be added before any of them are started. May be called from any thread.

void net_instaweb::RewriteContext::TracePrintf ( const char *  fmt,
  ... 
) [protected]

A convenience wrapper to log a trace annotation in both the request trace (if present) as well as the root user request trace (if present).

virtual GoogleString net_instaweb::RewriteContext::UserAgentCacheKey ( const ResourceContext *  context) const [inline, protected, virtual]

Indicates user agent capabilities that must be stored in the cache key.

Note that the context may be NULL as it may not be set before this. Since it isn't going to be modified in the method, ResourceContext is passed as a const pointer.

Reimplemented in net_instaweb::CssFilter::Context, and net_instaweb::InPlaceRewriteContext.


The documentation for this class was generated from the following file:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines