Page Speed Optimization Libraries  1.7.30.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines
Public Types | Public Member Functions | Static Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends
net_instaweb::SerfUrlAsyncFetcher Class Reference

#include "serf_url_async_fetcher.h"

Inheritance diagram for net_instaweb::SerfUrlAsyncFetcher:
net_instaweb::UrlAsyncFetcher

List of all members.

Public Types

enum  WaitChoice { kThreadedOnly, kMainlineOnly, kThreadedAndMainline }

Public Member Functions

 SerfUrlAsyncFetcher (const char *proxy, apr_pool_t *pool, ThreadSystem *thread_system, Statistics *statistics, Timer *timer, int64 timeout_ms, MessageHandler *handler)
 SerfUrlAsyncFetcher (SerfUrlAsyncFetcher *parent, const char *proxy)
virtual void ShutDown ()
virtual bool SupportsHttps () const
virtual void Fetch (const GoogleString &url, MessageHandler *message_handler, AsyncFetch *callback)
int Poll (int64 max_wait_ms)
bool WaitForActiveFetches (int64 max_milliseconds, MessageHandler *message_handler, WaitChoice wait_choice)
void FetchComplete (SerfFetch *fetch)
void ReportCompletedFetchStats (SerfFetch *fetch)
 Update the statistics object with results of the (completed) fetch.
apr_pool_t * pool () const
serf_context_t * serf_context () const
void PrintActiveFetches (MessageHandler *handler) const
virtual int64 timeout_ms ()
ThreadSystem * thread_system ()
void set_list_outstanding_urls_on_error (bool x)
bool track_original_content_length () const
void set_track_original_content_length (bool x)
void set_inflation_content_type_blacklist (const std::set< const ContentType * > &bypass_set)
bool SetHttpsOptions (StringPiece directive)
void SetSslCertificatesDir (StringPiece dir)
const GoogleString & ssl_certificates_dir () const
void SetSslCertificatesFile (StringPiece file)
const GoogleString & ssl_certificates_file () const

Static Public Member Functions

static void InitStats (Statistics *statistics)
static bool ValidateHttpsOptions (StringPiece directive, GoogleString *error_message)

Protected Types

typedef Pool< SerfFetchSerfFetchPool
typedef std::vector< SerfFetch * > FetchVector

Protected Member Functions

bool allow_https () const
 Determines whether https is allowed in the current configuration.
bool allow_self_signed () const
bool allow_unknown_certificate_authority () const
bool allow_certificate_not_yet_valid () const
void set_https_options (uint32 https_options)
void Init (apr_pool_t *parent_pool, const char *proxy)
bool SetupProxy (const char *proxy)
bool StartFetch (SerfFetch *fetch)
virtual bool AnyPendingFetches ()
int ApproximateNumActiveFetches ()
void CancelActiveFetches ()
void CancelActiveFetchesMutexHeld ()
bool WaitForActiveFetchesHelper (int64 max_ms, MessageHandler *message_handler)
void CleanupFetchesWithErrors ()
bool shutdown () const
 These must be accessed with mutex_ held.
void set_shutdown (bool s)

Protected Attributes

apr_pool_t * pool_
ThreadSystem * thread_system_
Timer * timer_
ThreadSystem::CondvarCapableMutex * mutex_
 mutex_ protects serf_context_ and active_fetches_.
serf_context_t * serf_context_
SerfFetchPool active_fetches_
SerfFetchPool completed_fetches_
SerfThreadedFetcher * threaded_fetcher_
Variable * active_count_

Friends

class SerfFetch
 To access stats variables below.

Detailed Description

Todo:
TODO(sligocki): Serf does not seem to act appropriately in IPv6 environments, fix and test this. Specifically: (1) It does not attempt to fall-back to IPv4 if IPv6 connection fails; (2) It may not correctly signal failure, which causes the incoming connection to hang.

Member Function Documentation

virtual bool net_instaweb::SerfUrlAsyncFetcher::AnyPendingFetches ( ) [protected, virtual]

AnyPendingFetches is accurate only at the time of call; this is used conservatively during shutdown. It counts fetches that have been requested by some thread, and can include fetches for which no action has yet been taken (ie fetches that are not active).

ApproximateNumActiveFetches can under- or over-count and is used only for error reporting.

This cleans up the serf resources for fetches that errored out. Must be called only immediately after running the serf event loop. Must be called with mutex_ held.

virtual void net_instaweb::SerfUrlAsyncFetcher::Fetch ( const GoogleString &  url,
MessageHandler *  message_handler,
AsyncFetch fetch 
) [virtual]

Asynchronously fetch a URL, set the response headers and stream the contents to fetch and call fetch->Done() when the fetch finishes.

There is an unchecked contract that response_headers are set before the response_writer or callback are used. Caution, several implementations do not satisfy this contract (but should).

Todo:

TODO(sligocki): GoogleString -> GoogleUrl or at least StringPiece.

TODO(sligocki): Include the URL in the fetch, like the request headers.

Implements net_instaweb::UrlAsyncFetcher.

Remove the completed fetch from the active fetch set, and put it into a completed fetch list to be cleaned up.

int net_instaweb::SerfUrlAsyncFetcher::Poll ( int64  max_wait_ms)
Todo:
TODO(morlovich): Make private once non-thread mode concept removed.

Indicates that Serf should enumerate failing URLs whenever the underlying Serf library reports an error.

bool net_instaweb::SerfUrlAsyncFetcher::SetHttpsOptions ( StringPiece  directive)

Indicates that direct HTTPS fetching should be allowed, and how picky to be about certificates. The directive is a comma separated list of these keywords: enable disable allow_self_signed allow_unknown_certificate_authority allow_certificate_not_yet_valid Returns 'false' if the directive does not parse properly.

virtual void net_instaweb::SerfUrlAsyncFetcher::ShutDown ( ) [virtual]

Stops all active fetches and prevents further fetches from starting (they will instead quickly call back to ->Done(false).

Reimplemented from net_instaweb::UrlAsyncFetcher.

Start a SerfFetch. Takes ownership of fetch and makes sure callback is called even if fetch fails to start.

mutex_ must be held before calling StartFetch.

virtual bool net_instaweb::SerfUrlAsyncFetcher::SupportsHttps ( ) const [virtual]

Determine if the fetcher supports fetching using HTTPS. By default we assume a fetcher can.

Reimplemented from net_instaweb::UrlAsyncFetcher.

virtual int64 net_instaweb::SerfUrlAsyncFetcher::timeout_ms ( ) [inline, virtual]

Returns a maximum time that we will allow fetches to take, or kUnspecifiedTimeout (the default) if we don't promise to timeout fetches.

Reimplemented from net_instaweb::UrlAsyncFetcher.

Indicates that Serf should track the original content length for fetched resources.

static bool net_instaweb::SerfUrlAsyncFetcher::ValidateHttpsOptions ( StringPiece  directive,
GoogleString *  error_message 
) [inline, static]

Validates the correctness of an https directive. Exposed as a static method for early exit on mis-specified pagespeed.conf.


Member Data Documentation

This is protected because it's updated along with active_fetches_, which happens in subclass SerfThreadedFetcher as well as this class.


The documentation for this class was generated from the following file:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines