Page Speed Optimization Libraries  1.9.32.2
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Namespaces | Classes | Typedefs | Enumerations | Functions | Variables
net_instaweb Namespace Reference

Unit-test framework for wget fetcher. More...

Namespaces

 css_util
 

Classes

class  ApacheConfig
 
class  ApacheMessageHandler
 
class  ApacheRequestContext
 
class  ApacheRewriteDriverFactory
 Creates an Apache RewriteDriver. More...
 
class  ApacheServerContext
 
class  ApacheThreadSystem
 
class  ApacheWriter
 Writer object that writes to an Apache Request stream. More...
 
class  AprFileSystem
 
class  AprTimer
 
class  InstawebContext
 
class  ApacheFetch
 
class  InstawebHandler
 
class  ModSpdyFetchController
 
class  ModSpdyFetcher
 
class  CacheHtmlFlow
 
class  FlushEarlyFlow
 
class  HtmlDetector
 
class  ProxyFetchFactory
 
class  ProxyFetchPropertyCallback
 
class  ProxyFetchPropertyCallbackCollector
 
class  ProxyFetch
 
class  ProxyInterface
 
class  ProxyUrlNamer
 
class  MockFilter
 
class  CreateFilterCallback
 
class  BackgroundFetchCheckingAsyncFetch
 
class  BackgroundFetchCheckingUrlAsyncFetcher
 
class  ProxyInterfaceTestBase
 
class  FileRewriter
 
class  StaticRewriter
 
class  RewriteOptionsManager
 
class  FileDriver
 
class  FileStatisticsLog
 Statistics logger that sends its output to a file. More...
 
class  LoggingFilter
 Counts some basic statistics observed as HTML is parsed. More...
 
class  StatisticsLog
 
class  AsyncFetch
 
class  StringAsyncFetch
 
class  AsyncFetchUsingWriter
 
class  SharedAsyncFetch
 
class  FallbackSharedAsyncFetch
 
class  ConditionalSharedAsyncFetch
 
class  AsyncFetchWithLock
 
class  CacheUrlAsyncFetcher
 
class  CountingUrlAsyncFetcher
 
class  ExternalUrlFetcher
 
class  FetcherTest
 
class  HTTPCache
 
class  HttpDumpUrlAsyncWriter
 
class  HttpDumpUrlFetcher
 
class  HttpResponseParser
 
class  HTTPValue
 
class  HTTPValueWriter
 
class  InflatingFetch
 
class  ExpectStringAsyncFetch
 
class  MockUrlFetcher
 
class  RateController
 
class  RateControllingUrlAsyncFetcher
 
class  ReflectingTestFetcher
 
class  SimulatedDelayFetcher
 
class  SyncFetcherAdapterCallback
 Class to help run an asynchronous fetch synchronously with a timeout. More...
 
class  UserAgentSensitiveTestFetcher
 
class  UrlAsyncFetcher
 
class  UrlAsyncFetcherStats
 
class  WaitUrlAsyncFetcher
 
class  WgetUrlFetcher
 
class  WriteThroughHTTPCache
 Composes two cache interfaces to form a two level http cache. More...
 
class  ImageTestingPeer
 
class  AddHeadFilter
 
class  AddInstrumentationFilter
 Injects javascript instrumentation for monitoring page-rendering time. More...
 
class  AssociationTransformer
 
class  AssociationSlot
 
class  BaseTagFilter
 
class  BeaconCriticalImagesFinder
 
class  BeaconCriticalLineInfoFinder
 
class  CacheExtender
 
class  CacheHtmlFilter
 
class  CacheHtmlInfoFinder
 Manages the cache lifetimes of CacheHtmlInfo. More...
 
class  CacheableResourceBase
 
class  CollectFlushEarlyContentFilter
 
class  CommonFilter
 
class  ComputeVisibleTextFilter
 
class  CriticalCssBeaconFilter
 
class  CriticalCssFilter
 
class  CriticalCssFinder
 Finds critical CSS rules (i.e. CSS needed for the initial page load). More...
 
struct  BeaconMetadata
 
class  CriticalImagesBeaconFilter
 
struct  CriticalImagesInfo
 
class  CriticalImagesFinder
 
class  TestCriticalImagesFinder
 
class  CriticalImagesFinderTestBase
 
class  CriticalLineInfoFinder
 
class  CriticalSelectorFilter
 
struct  CriticalSelectorInfo
 
class  CriticalSelectorFinder
 
class  BeaconCriticalSelectorFinder
 
class  CssAbsolutify
 
class  CssCombineFilter
 
class  CssFilter
 
class  CssFlattenImportsContext
 Context used by CssFilter under async flow that flattens . More...
 
class  CssHierarchy
 
class  CssImageRewriter
 
class  CssInlineFilter
 Inline small CSS files. More...
 
class  CssInlineImportToLinkFilter
 
class  CssMinify
 
class  CssMoveToHeadFilter
 
class  CssOutlineFilter
 Filter to take explicit <style> and <script> tags and outline them to files. More...
 
class  CssResourceSlot
 A place storing a rewritable URL inside a CSS AST. More...
 
class  CssResourceSlotFactory
 
class  CssRewriteTestBase
 
class  CssSummarizerBase
 
class  CssTagScanner
 
class  RewriteDomainTransformer
 
class  CssUrlCounter
 
class  CssUrlEncoder
 
class  CssUrlExtractor
 
class  CustomRewriteTestBase
 
class  DataUrlInputResource
 
class  DebugFilter
 
class  DecisionTree
 
class  DecodeRewrittenUrlsFilter
 
class  DedupInlinedImagesFilter
 
class  DeferIframeFilter
 
class  DelayImagesFilter
 
class  DeterministicJsFilter
 
class  DeviceProperties
 
class  DomStatsFilter
 Counts some basic statistics observed as HTML is parsed. More...
 
class  DomainLawyer
 
class  DomainRewriteFilter
 
class  DownstreamCachePurger
 
class  DownstreamCachingDirectives
 
class  ExperimentMatcher
 
class  FakeFilter
 
class  FileInputResource
 
class  FileLoadMapping
 
class  FileLoadMappingRegexp
 
class  FileLoadMappingLiteral
 
class  FileLoadPolicy
 
class  FileLoadRule
 
class  FileLoadRuleRegexp
 
class  FileLoadRuleLiteral
 
class  FixReflowFilter
 
class  FlushEarlyContentWriterFilter
 
class  FlushEarlyInfoFinder
 
class  MeaningfulFlushEarlyInfoFinder
 
class  FlushHtmlFilter
 
class  ScriptEditor
 Edit a substring in a script element. More...
 
class  GoogleAnalyticsFilter
 
class  GoogleFontCssInlineFilter
 
class  GoogleFontServiceInputResource
 
class  HandleNoscriptRedirectFilter
 
class  Image
 
class  ImageCombineFilter
 
class  ImageRewriteFilter
 
class  ImageTestBase
 
class  ImageUrlEncoder
 
class  InPlaceRewriteResourceSlot
 
class  InPlaceRewriteContext
 Context that is used for an in-place rewrite. More...
 
class  RecordingFetch
 
class  InlineRewriteContext
 
class  InsertDnsPrefetchFilter
 
class  InsertGAFilter
 
class  JavascriptRewriteConfig
 
class  JavascriptCodeBlock
 
class  JavascriptFilter
 
class  JavascriptSourceMapFilter
 
class  JavascriptLibraryIdentification
 
class  JsCombineFilter
 
class  JsDeferDisabledFilter
 
class  JsDisableFilter
 
class  JsInlineFilter
 Inline small Javascript files. More...
 
class  JsOutlineFilter
 Filter to take explicit <style> and <script> tags and outline them to files. More...
 
class  LazyloadImagesFilter
 
class  LocalStorageCacheFilter
 
class  MakeShowAdsAsyncFilter
 
class  MetaTagFilter
 
struct  ElementSample
 
class  MobilizeLabelFilter
 
struct  MobileRole
 A mobile role and its associated HTML attribute value. More...
 
class  MobilizeRewriteFilter
 
class  MockCriticalCssFinder
 
class  MockCriticalImagesFinder
 
class  MockResourceCallback
 
class  OutputResource
 
class  PedanticFilter
 
class  ProcessContext
 
class  RedirectOnSizeLimitFilter
 
class  RequestProperties
 
class  Resource
 
struct  TimedBool
 A boolean with an expiration date. More...
 
class  ResourceCombiner
 
class  ResourceFetch
 
class  ResourceNamer
 
class  ResourceSlot
 
class  FetchResourceSlot
 
class  HtmlResourceSlot
 
class  HtmlResourceSlotComparator
 
class  RewriteContext
 
class  TrimWhitespaceRewriter
 Simple test filter just trims whitespace from the input resource. More...
 
class  TrimWhitespaceSyncFilter
 
class  UpperCaseRewriter
 A similarly structured test-filter: this one just upper-cases its text. More...
 
class  NestedFilter
 
class  CombiningFilter
 
class  RewriteContextTestBase
 
class  RewriteDriver
 
class  OptionsAwareHTTPCacheCallback
 
class  RewriteDriverFactory
 
class  RewriteDriverPool
 
class  RewriteFilter
 
class  RewriteGflags
 
class  RewriteOptions
 
class  RewriteOptionsTestBase
 
class  RewriteQuery
 
class  RewriteStats
 Collects a few specific statistics variables related to Rewriting. More...
 
class  RewriteTestBase
 
class  RewrittenContentScanningFilter
 
class  ScanFilter
 
class  ScriptTagScanner
 
class  ServerContext
 
class  SimpleTextFilter
 
class  SingleRewriteContext
 
class  SplitHtmlBeaconFilter
 Inject JavaScript for detecting the below-the-fold HTML panels. More...
 
struct  XpathUnit
 
class  SplitHtmlConfig
 
class  SplitHtmlState
 Represents the filter state necessary to perform the split. More...
 
class  SplitHtmlFilter
 
class  SplitHtmlHelperFilter
 
class  StaticAssetManager
 
class  StripNonCacheableFilter
 
class  StripScriptsFilter
 Remove all scripts from a web page. More...
 
class  SupportNoscriptFilter
 
class  SuppressPreheadFilter
 
class  TestDistributedFetcher
 
class  TestRewriteDriverFactory
 
class  TestUrlNamer
 
class  UrlInputResource
 
class  UrlLeftTrimFilter
 
class  UrlNamer
 
class  UrlPartnership
 
class  UsageDataReporter
 
class  AddHeadersFetcher
 
class  AdminSite
 Implements the /pagespeed_admin pages. More...
 
class  AprMemCache
 
class  InPlaceResourceRecorder
 
class  LoopbackRouteFetcher
 See file comment. More...
 
struct  SerfStats
 
class  SerfUrlAsyncFetcher
 
class  SystemCachePath
 
class  SystemCaches
 
class  SystemMessageHandler
 
class  SystemRequestContext
 
class  SystemRewriteDriverFactory
 A server context with features specific to a psol port on a unix system. More...
 
class  SystemRewriteOptions
 
class  SystemServerContext
 A server context with features specific to a PSOL port on a unix system. More...
 
class  SystemThreadSystem
 

Typedefs

typedef Callback2< StringPiece,
bool * > 
HeaderPredicateFn
 
typedef std::map< GoogleString,
const Panel * > 
PanelIdToSpecMap
 
typedef std::multimap
< GoogleString, std::pair
< GoogleString, const int >
, StringCompareInsensitive > 
AttributesToNonCacheableValuesMap
 
typedef std::map< GoogleString,
std::pair< int32, int32 > > 
RenderedImageDimensionsMap
 
typedef RefCountedPtr
< CssResourceSlot
CssResourceSlotPtr
 
typedef RefCountedPtr< ResourceResourcePtr
 
typedef std::vector< ResourcePtr > ResourceVector
 
typedef Callback2< const
GoogleUrl &, bool * > 
ResourceUrlClaimant
 
typedef RefCountedPtr
< ResourceSlot
ResourceSlotPtr
 
typedef RefCountedPtr
< HtmlResourceSlot
HtmlResourceSlotPtr
 
typedef std::vector
< ResourceSlotPtr > 
ResourceSlotVector
 
typedef std::set
< HtmlResourceSlotPtr,
HtmlResourceSlotComparator
HtmlResourceSlotSet
 
typedef RefCountedPtr
< OutputResource
OutputResourcePtr
 
typedef std::vector
< OutputResourcePtr > 
OutputResourceVector
 
typedef std::vector< XpathUnitXpathUnits
 
typedef std::map< GoogleString,
XpathUnits * > 
XpathMap
 Map of xpath to XpathUnits.
 
typedef bool(* WebpProgressHook )(int percent, void *user_data)
 Progress hook for WebP conversions.
 

Enumerations

enum  BeaconStatus { kDoNotBeacon, kBeaconNoNonce, kBeaconWithNonce }
 
enum  InlineResult {
  INLINE_SUCCESS, INLINE_UNSUPPORTED_DEVICE, INLINE_NOT_CRITICAL, INLINE_NO_DATA,
  INLINE_TOO_LARGE, INLINE_CACHE_SMALL_IMAGES_UNREWRITTEN, INLINE_INTERNAL_ERROR
}
 
enum  MobileRelevantTag {
  kATag = 0, kArticleTag, kAsideTag, kButtonTag,
  kContentTag, kDatalistTag, kDivTag, kFieldsetTag,
  kFooterTag, kFormTag, kH1Tag, kH2Tag,
  kH3Tag, kH4Tag, kH5Tag, kH6Tag,
  kHeaderTag, kImgTag, kInputTag, kLegendTag,
  kMainTag, kMenuTag, kNavTag, kOptgroupTag,
  kOptionTag, kPTag, kSectionTag, kSelectTag,
  kSpanTag, kTextareaTag, kNumRelevantTags
}
 
enum  MobileAttrSubstring {
  kArticleAttr = 0, kAsideAttr, kBodyAttr, kBottomAttr,
  kCenterAttr, kColumnAttr, kCommentAttr, kContentAttr,
  kFindAttr, kFootAttr, kHdrAttr, kHeadAttr,
  kLeftAttr, kLogoAttr, kMainAttr, kMarginAttr,
  kMenuAttr, kMiddleAttr, kNavAttr, kRightAttr,
  kSearchAttr, kSecAttr, kTitleAttr, kTopAttr,
  kNumAttrStrings
}
 
enum  FeatureName {
  kElementTagDepth = 0, kPreviousTagCount, kPreviousTagPercent, kPreviousContentBytes,
  kPreviousContentPercent, kPreviousNonBlankBytes, kPreviousNonBlankPercent, kContainedTagDepth,
  kContainedTagRelativeDepth, kContainedTagCount, kContainedTagPercent, kContainedContentBytes,
  kContainedContentPercent, kContainedNonBlankBytes, kContainedNonBlankPercent, kHasAttrString,
  kRelevantTagCount = kHasAttrString + kNumAttrStrings, kRelevantTagPercent = kRelevantTagCount + kNumRelevantTags, kNumFeatures = kRelevantTagPercent + kNumRelevantTags
}
 
enum  OutputResourceKind { kRewrittenResource, kOnTheFlyResource, kOutlinedResource }
 
enum  PropertyCacheDecodeResult { kPropertyCacheDecodeNotFound, kPropertyCacheDecodeExpired, kPropertyCacheDecodeParseError, kPropertyCacheDecodeOk }
 
enum  PropertyCacheUpdateResult { kPropertyCacheUpdateNotFound, kPropertyCacheUpdateEncodeError, kPropertyCacheUpdateOk }
 
enum  RewriteResult { kRewriteFailed, kRewriteOk, kTooBusy }
 

Functions

void AprReportError (MessageHandler *message_handler, const char *filename, int line, const char *message, int error_code)
 
void ApacheRequestToRequestHeaders (const request_rec &request, RequestHeaders *request_headers, HeaderPredicateFn *predicate)
 
void ApacheRequestToRequestHeaders (const request_rec &request, RequestHeaders *request_headers)
 Fully converts apache request header structure into RequestHeaders.
 
void ApacheRequestToResponseHeaders (const request_rec &request, ResponseHeaders *headers, ResponseHeaders *err_headers)
 
void ResponseHeadersToApacheRequest (const ResponseHeaders &response_headers, request_rec *request)
 
void ErrorHeadersToApacheRequest (const ResponseHeaders &err_response_headers, request_rec *request)
 
void DisableDownstreamHeaderFilters (request_rec *request)
 Remove downstream filters that might corrupt our caching headers.
 
void PrintHeaders (request_rec *request)
 Debug utility for printing Apache headers to stdout.
 
void DisableCachingRelatedHeaders (request_rec *request)
 Updates headers related to caching (but not Cache-Control).
 
void DisableCacheControlHeader (request_rec *request)
 
template<class T >
apr_status_t apache_cleanup (void *object)
 Generic deleter meant to be used with apr_pool_cleanup_register().
 
void attach_mod_spdy ()
 Needs to be called from a ap_hook_optional_fn_retrieve hook.
 
int mod_spdy_get_spdy_version (conn_rec *conn)
 
spdy_slave_connection_factory * mod_spdy_create_slave_connection_factory (conn_rec *master_connection)
 
void mod_spdy_destroy_slave_connection_factory (spdy_slave_connection_factory *factory)
 
spdy_slave_connection * mod_spdy_create_slave_connection (spdy_slave_connection_factory *factory, ap_filter_rec_t *input_filter, void *input_filter_ctx, ap_filter_rec_t *output_filter, void *output_filter_ctx)
 
void mod_spdy_run_slave_connection (spdy_slave_connection *conn)
 
void mod_spdy_destroy_slave_connection (spdy_slave_connection *conn)
 
bool mod_ssl_is_https (conn_rec *conn)
 
bool ValidateAndExpireNonce (int64 now_ms, StringPiece nonce, CriticalKeys *critical_keys)
 
void GetCriticalKeysFromProto (int64 support_percentage, const CriticalKeys &critical_keys, StringSet *keys)
 
void UpdateCriticalKeys (bool require_prior_support, const StringSet &new_set, int support_value, CriticalKeys *critical_keys)
 
bool ShouldBeacon (const CriticalKeys &proto, const RewriteDriver &driver)
 
void WriteCriticalKeysToPropertyCache (const StringSet &new_keys, StringPiece nonce, int support_interval, bool should_replace_prior_result, bool require_prior_support, StringPiece property_name, const PropertyCache *cache, const PropertyCache::Cohort *cohort, AbstractPropertyPage *page, MessageHandler *message_handler, Timer *timer)
 
void PrepareForBeaconInsertionHelper (CriticalKeys *proto, NonceGenerator *nonce_generator, RewriteDriver *driver, bool using_candidate_key_detection, BeaconMetadata *result)
 
bool UpdateCandidateKeys (const StringSet &keys, CriticalKeys *proto, bool clear_rebeacon_timestamp)
 
bool IsBeaconDataAvailable (const CriticalKeys &proto)
 
ImageNewImage (const StringPiece &original_contents, const GoogleString &url, const StringPiece &file_prefix, Image::CompressionOptions *options, Timer *timer, MessageHandler *handler)
 
ImageBlankImageWithOptions (int width, int height, ImageType type, const StringPiece &tmp_dir, Timer *timer, MessageHandler *handler, Image::CompressionOptions *options)
 
int CharToInt (char c)
 char to int without sign extension. More...
 
int JpegIntAtPosition (const StringPiece &buf, size_t pos)
 
int GifIntAtPosition (const StringPiece &buf, size_t pos)
 
int PngIntAtPosition (const StringPiece &buf, size_t pos)
 
bool PngSectionIdIs (const char *hdr, const StringPiece &buf, size_t pos)
 
const PropertyValue * DecodeFromPropertyCacheHelper (const PropertyCache *cache, AbstractPropertyPage *page, const PropertyCache::Cohort *cohort, StringPiece property_name, int64 cache_ttl_ms, PropertyCacheDecodeResult *status)
 
template<typename T >
T * DecodeFromPropertyCache (const PropertyCache *cache, AbstractPropertyPage *page, const PropertyCache::Cohort *cohort, StringPiece property_name, int64 cache_ttl_ms, PropertyCacheDecodeResult *status)
 
template<typename T >
T * DecodeFromPropertyCache (RewriteDriver *driver, const PropertyCache::Cohort *cohort, StringPiece property_name, int64 cache_ttl_ms, PropertyCacheDecodeResult *status)
 
PropertyCacheUpdateResult UpdateInPropertyCache (const protobuf::MessageLite &value, const PropertyCache::Cohort *cohort, StringPiece property_name, bool write_cohort, AbstractPropertyPage *page)
 
PropertyCacheUpdateResult UpdateInPropertyCache (const protobuf::MessageLite &value, RewriteDriver *driver, const PropertyCache::Cohort *cohort, StringPiece property_name, bool write_cohort)
 
bool OptimizeWebp (const GoogleString &original_jpeg, int configured_quality, WebpProgressHook progress_hook, void *progress_hook_data, GoogleString *compressed_webp, MessageHandler *message_handler)
 
bool ReduceWebpImageQuality (const GoogleString &original_webp, int quality, GoogleString *compressed_webp)
 
apr_pool_t * AprCreateThreadCompatiblePool (apr_pool_t *parent_pool)
 

Variables

const char kPagespeedOriginalUrl [] = "mod_pagespeed_original_url"
 
const char kModPagespeedFilterName [] = "MOD_PAGESPEED_OUTPUT_FILTER"
 Filter used for HTML rewriting.
 
const char kModPagespeedFixHeadersName [] = "MOD_PAGESPEED_FIX_HEADERS_FILTER"
 Filter used to fix headers after mod_headers runs.
 
const char kModPagespeedInPlaceFilterName [] = "MOD_PAGESPEED_IN_PLACE_FILTER"
 
const char kModPagespeedInPlaceCheckHeadersName []
 Second filter checks headers for cacheability. More...
 
const char kPageUrl [] = "page.html"
 
const char kBackgroundFetchHeader [] = "X-Background-Fetch"
 
const char kGaSnippetGetTracker [] = "_modpagespeed_getRewriteTracker("
 
const char kGaSnippetPrefix []
 
const char kGaSnippetSuffix []
 
const int64 kBeaconTimeoutIntervalMs = Timer::kMinuteMs
 
const int64 kHighFreqBeaconCount = 3
 
const int64 kLowFreqBeaconMult = 100
 
const int64 kNonceExpirationLimit = 5
 
const char kGAExperimentSnippet []
 
const char kGAJsSnippet []
 
const char kGASpeedTracking []
 
const
net_instaweb::DecisionTree::Node 
kNavigationalTree []
 
const double kNavigationalTreeThreshold
 
const int kNavigationalTreeSize
 
const
net_instaweb::DecisionTree::Node 
kHeaderTree []
 
const double kHeaderTreeThreshold
 
const int kHeaderTreeSize
 
const
net_instaweb::DecisionTree::Node 
kContentTree []
 
const double kContentTreeThreshold
 
const int kContentTreeSize
 
const int kNoQualityGiven = -1
 

Detailed Description

Unit-test framework for wget fetcher.

extern "C"

Base class for tests which want a ServerContext.

for NULL

for StringSet, etc

Some common routines and constants for tests dealing with Images.

for GoogleString

A filter which does not modify the DOM, but counts statistics about it.

Base class for tests which do rewrites within CSS.

namespace Css

for ResourcePtr

DO NOT EDIT. Generated by ./google_analytics_snippet_gen.py.

Callbacks used for testing.

for size_t

for FILE

for apr_status_t The httpd header must be after the instaweb_context.h. Otherwise, the compiler will complain "strtoul_is_not_a_portable_function_use_strtol_instead".

html_filter that passes data through unmodified, but logs statistics about the data as it goes by. It should be possible to create many instances of this class and insert them at different points in the rewriting flow Goal is to log: NUM_EXPLICIT_CLOSED - <tag> </tag> pairs NUM_IMPLICIT_CLOSED - <tag> for implicitly-closed tag NUM_BRIEF_CLOSED - </tag> NUM_CLOSED - Sum of above three NUM_UNCLOSED - <tag> without matching </tag> NUM_SPURIOUS_CLOSED - </tag> without preceding <tag>; UNCOUNTED RIGHT NOW! NUM_TAGS - Total number of opening tags NUM_CDATA - cdata sections NUM_COMMENTS - comments NUM_DIRECTIVES - directives NUM_DOCUMENTS - started documents NUM_IE_DIRECTIVES - ie directives Reporting: We report this information via a StatisticsLog: filter.ToString(log) Two sets of statistics (eg before and after processing) can be compared using before.Equals(after),

Todo:
TODO(sligocki): Find a way to forward declare FileSystem::InputFile.

Copyright 2011 Google Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http:///www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Helper class to make RewriteTestBase tests that use a custom options subclass.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Search for synchronous loads of Google Analytics similar to the following:

<script type="text/javascript">
    var gaJsHost = (("https:" == document.location.protocol) ?
        "https://ssl." : "http://www.");
    document.write(unescape("%3Cscript src='" + gaJsHost +
        "google-analytics.com/ga.js type='text/javascript'" +
        "%3E%3C/script%3E"));
</script>
<script type="text/javascript">
    try {
        var pageTracker = _gat._getTracker("UA-XXXXX-X");
        pageTracker._trackPageview();
    } catch(err) {}
</script>

Replace the document.write with a new snippet that loads ga.js asynchronously. Also, insert a replacement for _getTracker that converts any calls to the synchronous API to the asynchronous API. The _getTracker replacement is a new function that returns a mock tracker object. Anytime a synchronous API method is called, the mock tracker fowards it to a _gaq.push(...) call.

An alternative approach would been to find all the API calls and rewrite them to the asynchronous API. However, to be done properly, it would have had the added complication of using a JavaScript compiler.

Unit-test the RewriteContext class. This is made simplest by setting up some dummy rewriters in our test framework.

for pair

We need to include rewrite_driver.h due to covariant return of html_parse() We need to include mock_timer.h to allow upcast to Timer*.

Todo:
TODO(skerner): #include image_spriter.pb.h is needed to allow use of enum ImageFormat. Break this dependency and remove the include.
Todo:
TODO(huibao): Move LibpngImageLibrary to pagespeed/kernel/image/. Refactor LibpngImageLibrary. This class can be split into 3 parts: reader, writer, and canvas creater. The first two parts can be merged with png_optimizer.cc and png_optimizer.h.

This class is a simple wrapper around another fetcher that adds headers to requests based on settings in the rewrite options before passing them on to the backend fetcher.

Typedef Documentation

typedef Callback2<StringPiece, bool*> net_instaweb::HeaderPredicateFn

Defines a predicate function used to select which request-headers to copy. The callback sets the bool* arg (.second) to true if it wants to include the header.

The StringPiece is the name of the header.

typedef Callback2<const GoogleUrl&, bool*> net_instaweb::ResourceUrlClaimant

Sometimes some portions of URL space need to be handled differently by dedicated resource subclasses. ResourceProvider callbacks are used to teach RewriteDriver about these, so it knows not to build regular UrlInputResource objects.

Enumeration Type Documentation

Every feature has a symbolic name given by Name or Name + Index. DEFINITIONS OF FEATURES:

  • "Previous" features do not include the tag being labeled.
  • "Contained" and "Relevant" features do include the tag being labeled.
  • "TagCount" features ignore clearly non-user-visible tags such as <script>, <style>, and <link>, and include only tags inside <body>.
  • "TagDepth" features include only div-like tags such as
    , <section>, <header>, and <aside> (see kRoleTags and kDivLikeTags in mobilize_label_filter.cc). They are the nesting depth of the tag within <body>.
  • ElementTagDepth is the depth of the tag being sampled itself.
  • ContainedTagDepth is the maximum depth of any div-like child of this tag.
  • ContainedTagRelativeDepth is the difference between these two depths.
  • ContentBytes Ignores tags and their attributes, and also ignores leading and trailing whitespace between tags. So "hi there" is 8 ContentBytes, but "hi <i class='foo'>there</i>" is only 7 ContentBytes.
  • NonBlankBytes is like ContentBytes but ignores all whitespace.
  • HasAttrString is a family of 0/1 entries indicating whether the corresponding string (see kRelevantAttrSubstrings in mobilize_label_filter.cc) occurs in the class, id, or role attribute of the sampled tag.
  • RelevantTagCount is a series of counters indicating the number of various "interesting" HTML tags within the current tag. This includes all div-like tags along with tags such as
, ,

, and (see kRelevantTags in mobilize_label_filter.cc).

Attribute substrings that are considered interesting if they occur in the id, class, or role of a div-like tag.

The following three enums are used to name signals used by the decision trees and computed by the mobilize_label_filter. Tags that are considered relevant and are counted in a sample. Some tags are role tags or otherwise considered div-like. These tag names are used to index the RelevantTagCount and RelevantTagPercent features below. Note that it's possible to add new tags to this list.

Enumerator
kRewrittenResource 

derived from some input resource URL or URLs.

kOnTheFlyResource 

derived from some input resource URL or URLs in a very inexpensive way — it makes no sense to cache the output contents.

kOutlinedResource 

derived from page HTML.

Enumerator
kPropertyCacheDecodeNotFound 

includes property cache not being enabled.

Enumerator
kPropertyCacheUpdateNotFound 

can't find existing value to update

Used to signal whether optimization was successful or not to RewriteContext::RewriteDone.

Enumerator
kRewriteFailed 

rewrite is impossible or undesirable

kRewriteOk 

rewrite went fine

kTooBusy 

the system is temporarily too busy to handle this rewrite request; no conclusion can be drawn on whether it's worth trying again or not.

Function Documentation

void net_instaweb::ApacheRequestToRequestHeaders ( const request_rec &  request,
RequestHeaders *  request_headers,
HeaderPredicateFn *  predicate 
)

Converts Apache header structure into RequestHeaders, selecting only those for which the predicate sets its bool* argument to true. If the predicate is NULL, then all the headers are transferred.

The predicate should be created with NewPermanentCallback and stored in a scoped_ptr<Callback2>, so that it is deleted after this function completes.

void net_instaweb::ApacheRequestToResponseHeaders ( const request_rec &  request,
ResponseHeaders *  headers,
ResponseHeaders *  err_headers 
)

Converts Apache header structure (request.headers_out) into ResponseHeaders headers. If err_headers is not NULL then request.err_headers_out is copied into it. In the event that headers == err_headers, the headers from request.err_headers_out will be appended to the list of headers, but no merging occurs.

apr_pool_t* net_instaweb::AprCreateThreadCompatiblePool ( apr_pool_t *  parent_pool)

Creates a pool that can be used in any thread, even when run in Apache prefork.

1) This method must be called from startup phase only 2) Each pool must be accessed only from a single thread (or otherwise have its access serialized) 3) Different pools returned by this function may be safely used concurrently. 4) It's OK to just use ap_pool_create to create child pools of this one from multiple threads; those will be re-entrant too (but pools created merely as children of Apache's pools will not be reentrant in prefork)

In short, pools returned by this method are not fully threadsafe, but at least they are not thread-hostile, which is what you get with apr_pool_create in Prefork.

Note: the above is all about the release version of the pool code, the checking one has some additional locking!

WARNING: you must not call apr_pool_clear on the returned pool. The returned pool can be used to create sub-pools that can be accessed in distinct threads, due to a mutex injected into the allocator. However, if you call apr_pool_clear on the returned pool, the allocator's mutex will be freed and the pointer to it will be dangling. Subsequent allocations are likely to crash.

Image* net_instaweb::BlankImageWithOptions ( int  width,
int  height,
ImageType  type,
const StringPiece &  tmp_dir,
Timer *  timer,
MessageHandler *  handler,
Image::CompressionOptions *  options 
)

Creates a blank image of the given dimensions and type. For now, this is assumed to be an 8-bit 4-channel image transparent image.

int net_instaweb::CharToInt ( char  c)
inline

char to int without sign extension.

The following four helper functions were moved here for testability. We ran into problems with sign extension under different compiler versions, and we'd like to catch regressions on that front in the future.

template<typename T >
T* net_instaweb::DecodeFromPropertyCache ( const PropertyCache *  cache,
AbstractPropertyPage *  page,
const PropertyCache::Cohort *  cohort,
StringPiece  property_name,
int64  cache_ttl_ms,
PropertyCacheDecodeResult *  status 
)

Decodes a protobuf of type T from the property named 'property_name' in the cohort 'cohort_name' in the given property cache, and makes sure it has not exceeded its TTL of 'cache_ttl_ms' (a value of -1 will disable this check).

*status will denote the decoding state; if it's kPropertyCacheDecodeOk then a pointer to a freshly allocated decoded proto is returned; otherwise NULL is returned.

*status set by helper

template<typename T >
T* net_instaweb::DecodeFromPropertyCache ( RewriteDriver *  driver,
const PropertyCache::Cohort *  cohort,
StringPiece  property_name,
int64  cache_ttl_ms,
PropertyCacheDecodeResult *  status 
)

Wrapper version of the above function that gets the property cache and the property page from the given driver.

const PropertyValue* net_instaweb::DecodeFromPropertyCacheHelper ( const PropertyCache *  cache,
AbstractPropertyPage *  page,
const PropertyCache::Cohort *  cohort,
StringPiece  property_name,
int64  cache_ttl_ms,
PropertyCacheDecodeResult *  status 
)

Returns PropertyValue object for given cohort and property name, setting *status and returning NULL if any errors were found.

void net_instaweb::DisableCacheControlHeader ( request_rec *  request)

Updates caching headers to ensure the resulting response is not cached. Removes any max-age specification, and adds max-age=0, no-cache.

void net_instaweb::ErrorHeadersToApacheRequest ( const ResponseHeaders &  err_response_headers,
request_rec *  request 
)

Converts ResponseHeaders into Apache request err_headers. This function does not alter the status code or the major/minor version of the Apache request.

void net_instaweb::GetCriticalKeysFromProto ( int64  support_percentage,
const CriticalKeys &  critical_keys,
StringSet *  keys 
)

Generate a list of the critical keys from a proto, storing it into keys. Takes into account legacy keys that may have been added before. A key is considered critical if its support is at least support_percentage of the maximum possible support value (which ramps up as beacon results arrive). When support_percentage = 0, any support is sufficient; when support_percentage = 100 all beacon results must support criticality.

bool net_instaweb::IsBeaconDataAvailable ( const CriticalKeys &  proto)
inline

Based on the CriticalKeys data seen so far, describe whether beacon metadata is available. This returns false until data is received.

spdy_slave_connection_factory* net_instaweb::mod_spdy_create_slave_connection_factory ( conn_rec *  master_connection)

See mod_spdy's slave_connection_api.h for description of the methods below. These are merely forwarding wrappers with some CHECKS. Note that this method will return NULL if the relevant mod_spdy methods weren't found registered with Apache. Others, however, will CHECK-fail (since there is no sensible way to call them if this method failed); except you can always safely mod_spdy_destroy_slave_connection_factory(NULL).

int net_instaweb::mod_spdy_get_spdy_version ( conn_rec *  conn)

If the connection is using SPDY with mod_spdy, returns the protocol version. Otherwise, returns 0.

bool net_instaweb::mod_ssl_is_https ( conn_rec *  conn)

Returns true if given connection is using HTTPS. (This is actually a mod_ssl function).

Image* net_instaweb::NewImage ( const StringPiece &  original_contents,
const GoogleString &  url,
const StringPiece &  file_prefix,
Image::CompressionOptions *  options,
Timer *  timer,
MessageHandler *  handler 
)

Image owns none of its inputs. All of the arguments to NewImage(...) (the original_contents in particular) must outlive the Image object itself. The intent is that an Image is created in a scoped fashion from an existing known resource.

The options should be set via Image::SetOptions after construction, before the image is used for anything but determining its natural dimension size.

Todo:
TODO(jmarantz): It would seem natural to fold the ImageOptions into the Image object itself.
bool net_instaweb::OptimizeWebp ( const GoogleString &  original_jpeg,
int  configured_quality,
WebpProgressHook  progress_hook,
void *  progress_hook_data,
GoogleString *  compressed_webp,
MessageHandler *  message_handler 
)

Optimizer in the style of pagespeed/kernel/image/jpeg_optimizer.h that creates a webp-formatted image in compressed_webp from the jpeg image in original_jpeg. Indicates failure by returning false, in which case compressed_webp may be filled with junk.

void net_instaweb::PrepareForBeaconInsertionHelper ( CriticalKeys *  proto,
NonceGenerator *  nonce_generator,
RewriteDriver *  driver,
bool  using_candidate_key_detection,
BeaconMetadata *  result 
)

Given a set of candidate critical keys, decide whether beaconing should take place. We should always beacon if there's new critical key data. Otherwise re-beaconing is based on a time and request interval, and 2 modes of beaconing frequency are supported. At first, beaconing occurs at a high frequency until we have collected kHighFreqBeaconCount beacons; after that, we transition into low frequency beaconing mode, where beaconing occurs less often. We also track the number of expired nonces since the last valid beacon was received to see if beaconing is set up correctly, and if it looks like it isn't, only do low frequency beaconing. Sets status and nonce appropriately in *result (nonce will be empty if no nonce is required). If candidate keys are not required, keys may be empty (but new candidate detection will not occur). If result->status != kDontBeacon, caller should write proto back to the property cache using UpdateInPropertyCache.

bool net_instaweb::ReduceWebpImageQuality ( const GoogleString &  original_webp,
int  quality,
GoogleString *  compressed_webp 
)

Reduce the quality of the webp image. Indicates failure by returning false. WebP quality varies from 1 to 100. Original image will be returned if input quality is <1.

void net_instaweb::ResponseHeadersToApacheRequest ( const ResponseHeaders &  response_headers,
request_rec *  request 
)

Converts the ResponseHeaders to the output headers. This function does not alter the status code or the major/minor version of the Apache request.

bool net_instaweb::UpdateCandidateKeys ( const StringSet &  keys,
CriticalKeys *  proto,
bool  clear_rebeacon_timestamp 
)

Update the candidate key set in proto. If new candidate keys are detected, they are inserted into proto with a support value of 0, and true is returned. Otherwise returns false. If clear_rebeacon_timestamp is set, the rebeacon timestamp field in the proto is cleared to force rebeaconing on the next request.

void net_instaweb::UpdateCriticalKeys ( bool  require_prior_support,
const StringSet &  new_set,
int  support_value,
CriticalKeys *  critical_keys 
)

Add support for new_set to existing support. The new_set should be obtained from a fully-validated beacon result – this means PrepareForBeaconInsertion should have been called if required, and the resulting nonce should have been checked. If require_prior_support then there must be an existing support entry (possibly 0) for new support to be registered.

PropertyCacheUpdateResult net_instaweb::UpdateInPropertyCache ( const protobuf::MessageLite &  value,
RewriteDriver *  driver,
const PropertyCache::Cohort *  cohort,
StringPiece  property_name,
bool  write_cohort 
)
inline

Updates the property 'property_name' in cohort 'cohort_name' of the property cache managed by the rewrite driver with the new value of the proto T. If 'write_cohort' is true, will also additionally write out the cohort to the cache backing.

bool net_instaweb::ValidateAndExpireNonce ( int64  now_ms,
StringPiece  nonce,
CriticalKeys *  critical_keys 
)

Check whether the given nonce is valid, invalidating any expired nonce entries we might encounter. To avoid the need to copy and clear the nonce list, we invalidate the entry used and any expired entries by clearing the nonce value and timestamp. These entries will be reused by AddNonceToCriticalSelectors.

void net_instaweb::WriteCriticalKeysToPropertyCache ( const StringSet &  new_keys,
StringPiece  nonce,
int  support_interval,
bool  should_replace_prior_result,
bool  require_prior_support,
StringPiece  property_name,
const PropertyCache *  cache,
const PropertyCache::Cohort *  cohort,
AbstractPropertyPage *  page,
MessageHandler *  message_handler,
Timer *  timer 
)

Update the property cache with a new set of keys. This will update the support value for the new keys. If require_prior_support is set, any keys that are not already present in the property cache will be ignored (to prevent spurious keys from being injected). Note that it only increases the support value for the new keys, it does not decay values that are not present. PrepareForBeaconInsertion should have been called previously if !should_replace_prior_result and nonces must be checked.

Variable Documentation

const int64 net_instaweb::kBeaconTimeoutIntervalMs = Timer::kMinuteMs

The amount of time after generating a nonce that we will accept it as valid. This keeps an attacker from accumulating large numbers of valid nonces to send many beacon responses at once.

const char net_instaweb::kGaSnippetSuffix[]
Initial value:
=
" ];\n"
" var pageTracker = {\n"
" _initData: function() {},\n"
" };\n"
" for (var i=pageTrackerMethodNames.length; i--;) {\n"
" var n = pageTrackerMethodNames[i];\n"
" pageTracker[n] = deferTrackerFunc(prefix + n);\n"
" }\n"
" return pageTracker;\n"
" };\n"
"\n"
" var ga = document.createElement('script');\n"
" ga.type = 'text/javascript'; ga.async = true;\n"
" ga.src = ('https:' == document.location.protocol ? 'https:///ssl' :\n"
" 'http:///www') + '.google-analytics.com/ga.js';\n"
" var s = document.getElementsByTagName('script')[0];\n"
" s.parentNode.insertBefore(ga, s);\n"
"})();\n"
const int64 net_instaweb::kHighFreqBeaconCount = 3

The number of valid beacons received that will switch from high frequency to low frequency beaconing.

const int64 net_instaweb::kLowFreqBeaconMult = 100

The multiplier to apply to RewriteOptions::beacon_reinstrument_time_sec() to determine the low frequency beaconing interval. For example, the default value rebeaconing value is 5 seconds, so we will rebeacon every 5 seconds in high frequency mode, and every 500 seconds (~8 minutes) in low frequency mode.

const char net_instaweb::kModPagespeedInPlaceCheckHeadersName[]
Initial value:
=
"MOD_PAGESPEED_IN_PLACE_CHECK_HEADERS_FILTER"

Second filter checks headers for cacheability.

const char net_instaweb::kModPagespeedInPlaceFilterName[] = "MOD_PAGESPEED_IN_PLACE_FILTER"

Filters used for In-Place Resource Optimization. First filter stores un-gzipped contents.

const net_instaweb::DecisionTree::Node net_instaweb::kNavigationalTree[]

Forward declarations for the machine-generated decision trees used by mobilize_label_filter to classify DOM elements.

const int64 net_instaweb::kNonceExpirationLimit = 5

The limit on the number of nonces that can expire before we stop trying to do high frequency beaconing. This is a signal that beacons are not configured correctly and so we drop into low frequency beaconing mode.

const int net_instaweb::kNoQualityGiven = -1

Quality taken when no quality is passed through flags or when no quality is retrieved from JpegUtils::GetImageQualityFromImage.