net_instaweb::SerfUrlAsyncFetcher Class Reference

#include "serf_url_async_fetcher.h"

Inheritance diagram for net_instaweb::SerfUrlAsyncFetcher:

Public Types
enum	WaitChoice { kThreadedOnly, kMainlineOnly, kThreadedAndMainline }

Public Member Functions
	SerfUrlAsyncFetcher (const char *proxy, apr_pool_t *pool, ThreadSystem *thread_system, Statistics *statistics, Timer *timer, int64 timeout_ms, MessageHandler *handler)
	SerfUrlAsyncFetcher (SerfUrlAsyncFetcher *parent, const char *proxy)
virtual void	ShutDown ()
virtual bool	SupportsHttps () const
virtual void	Fetch (const GoogleString &url, MessageHandler *message_handler, AsyncFetch *callback)
int	Poll (int64 max_wait_ms)
bool	WaitForActiveFetches (int64 max_milliseconds, MessageHandler *message_handler, WaitChoice wait_choice)
void	FetchComplete (SerfFetch *fetch)
void	ReportCompletedFetchStats (const SerfFetch *fetch)
	Update the statistics object with results of the (completed) fetch.
void	ReportFetchSuccessStats (SerfCompletionResult result, const ResponseHeaders *headers, const SerfFetch *fetch)
	Updates states used for success/failure monitoring.
apr_pool_t *	pool () const
void	PrintActiveFetches (MessageHandler *handler) const
virtual int64	timeout_ms ()
ThreadSystem *	thread_system ()
void	set_list_outstanding_urls_on_error (bool x)
bool	track_original_content_length () const
void	set_track_original_content_length (bool x)
bool	SetHttpsOptions (StringPiece directive)
void	SetSslCertificatesDir (StringPiece dir)
const GoogleString &	ssl_certificates_dir () const
void	SetSslCertificatesFile (StringPiece file)
const GoogleString &	ssl_certificates_file () const
Public Member Functions inherited from net_instaweb::UrlAsyncFetcher
void	set_fetch_with_gzip (bool x)
bool	fetch_with_gzip () const
AsyncFetch *	EnableInflation (AsyncFetch *fetch) const

Static Public Member Functions
static void	InitStats (Statistics *statistics)
static bool	ValidateHttpsOptions (StringPiece directive, GoogleString *error_message)

Protected Types
typedef Pool< SerfFetch >	SerfFetchPool
typedef std::vector< SerfFetch * >	FetchVector

Protected Member Functions
bool	allow_https () const
	Determines whether https is allowed in the current configuration.
bool	allow_self_signed () const
bool	allow_unknown_certificate_authority () const
bool	allow_certificate_not_yet_valid () const
void	set_https_options (uint32 https_options)
void	Init (apr_pool_t *parent_pool, const char *proxy) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool	SetupProxy (const char *proxy) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool	StartFetch (SerfFetch *fetch) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
virtual bool	AnyPendingFetches ()
int	ApproximateNumActiveFetches ()
void	CancelActiveFetches ()
void	CancelActiveFetchesMutexHeld () EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool	WaitForActiveFetchesHelper (int64 max_ms, MessageHandler *message_handler)
void	CleanupFetchesWithErrors () EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool	shutdown () const EXCLUSIVE_LOCKS_REQUIRED(mutex_)
void	set_shutdown (bool s) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
Protected Member Functions inherited from net_instaweb::UrlAsyncFetcher
	UrlAsyncFetcher ()

Protected Attributes
apr_pool_t *	pool_
ThreadSystem *	thread_system_
Timer *	timer_
ThreadSystem::CondvarCapableMutex *	mutex_
SerfFetchPool	completed_fetches_
SerfThreadedFetcher *	threaded_fetcher_
UpDownCounter *	active_count_

Friends
class	SerfFetch
	To access stats variables below.

Additional Inherited Members
Static Public Attributes inherited from net_instaweb::UrlAsyncFetcher
static const int64	kUnspecifiedTimeout

Detailed Description

Todo:

TODO(sligocki): Serf does not seem to act appropriately in IPv6 environments, fix and test this. Specifically: (1) It does not attempt to fall-back to IPv4 if IPv6 connection fails; (2) It may not correctly signal failure, which causes the incoming connection to hang.

Member Function Documentation

virtual bool net_instaweb::SerfUrlAsyncFetcher::AnyPendingFetches ( )

protectedvirtual

AnyPendingFetches is accurate only at the time of call; this is used conservatively during shutdown. It counts fetches that have been requested by some thread, and can include fetches for which no action has yet been taken (ie fetches that are not active).

int net_instaweb::SerfUrlAsyncFetcher::ApproximateNumActiveFetches ( )

protected

ApproximateNumActiveFetches can under- or over-count and is used only for error reporting.

void net_instaweb::SerfUrlAsyncFetcher::CleanupFetchesWithErrors ( )

protected

This cleans up the serf resources for fetches that errored out. Must be called only immediately after running the serf event loop.

virtual void net_instaweb::SerfUrlAsyncFetcher::Fetch ( const GoogleString &  url, MessageHandler *  message_handler, AsyncFetch *  fetch  )

virtual

Asynchronously fetch a URL, set the response headers and stream the contents to fetch and call fetch->Done() when the fetch finishes.

There is an unchecked contract that response_headers are set before the response_writer or callback are used. Caution, several implementations do not satisfy this contract (but should).

Todo:

TODO(sligocki): GoogleString -> GoogleUrl or at least StringPiece.

TODO(sligocki): Include the URL in the fetch, like the request headers.

Implements net_instaweb::UrlAsyncFetcher.

void net_instaweb::SerfUrlAsyncFetcher::FetchComplete

(

SerfFetch *

fetch

)

Remove the completed fetch from the active fetch set, and put it into a completed fetch list to be cleaned up.

int net_instaweb::SerfUrlAsyncFetcher::Poll

(

int64

max_wait_ms

)

Todo:

TODO(morlovich): Make private once non-thread mode concept removed.

void net_instaweb::SerfUrlAsyncFetcher::set_list_outstanding_urls_on_error

(

bool

x

)

Indicates that Serf should enumerate failing URLs whenever the underlying Serf library reports an error.

bool net_instaweb::SerfUrlAsyncFetcher::SetHttpsOptions

(

StringPiece

directive

)

Indicates that direct HTTPS fetching should be allowed, and how picky to be about certificates. The directive is a comma separated list of these keywords: enable disable allow_self_signed allow_unknown_certificate_authority allow_certificate_not_yet_valid Returns 'false' if the directive does not parse properly.

virtual void net_instaweb::SerfUrlAsyncFetcher::ShutDown ( )

virtual

Stops all active fetches and prevents further fetches from starting (they will instead quickly call back to ->Done(false).

Reimplemented from net_instaweb::UrlAsyncFetcher.

bool net_instaweb::SerfUrlAsyncFetcher::StartFetch ( SerfFetch *  fetch )

protected

Start a SerfFetch. Takes ownership of fetch and makes sure callback is called even if fetch fails to start.

virtual bool net_instaweb::SerfUrlAsyncFetcher::SupportsHttps ( ) const

virtual

Determine if the fetcher supports fetching using HTTPS. By default we assume a fetcher can.

Reimplemented from net_instaweb::UrlAsyncFetcher.

virtual int64 net_instaweb::SerfUrlAsyncFetcher::timeout_ms ( )

inlinevirtual

Returns a maximum time that we will allow fetches to take, or kUnspecifiedTimeout (the default) if we don't promise to timeout fetches.

Reimplemented from net_instaweb::UrlAsyncFetcher.

bool net_instaweb::SerfUrlAsyncFetcher::track_original_content_length ( ) const

inline

Indicates that Serf should track the original content length for fetched resources.

static bool net_instaweb::SerfUrlAsyncFetcher::ValidateHttpsOptions ( StringPiece  directive, GoogleString *  error_message  )

inlinestatic

Validates the correctness of an https directive. Exposed as a static method for early exit on mis-specified pagespeed.conf.

Member Data Documentation

UpDownCounter* net_instaweb::SerfUrlAsyncFetcher::active_count_

protected

This is protected because it's updated along with active_fetches_, which happens in subclass SerfThreadedFetcher as well as this class.

ThreadSystem::CondvarCapableMutex* net_instaweb::SerfUrlAsyncFetcher::mutex_

protected

mutex_ protects serf_context_, active_fetches_ and shutdown_. It's protected because SerfThreadedFetcher needs access.

---

The documentation for this class was generated from the following file:

• pagespeed/system/serf_url_async_fetcher.h