com.yahoo.search.federation.http

Class HTTPProviderSearcher

java.lang.Object

com.yahoo.component.AbstractComponent

com.yahoo.component.chain.ChainedComponent

com.yahoo.processing.Processor

com.yahoo.search.Searcher

com.yahoo.search.cluster.PingableSearcher

com.yahoo.search.cluster.ClusterSearcher<Connection>

com.yahoo.search.federation.http.HTTPSearcher

com.yahoo.search.federation.http.HTTPProviderSearcher

All Implemented Interfaces:

com.yahoo.component.Component, NodeManager<Connection>, Comparable<com.yahoo.component.Component>

Direct Known Subclasses:

ConfiguredHTTPProviderSearcher

public abstract class HTTPProviderSearcher
extends HTTPSearcher

Superclass of searchers which talks to HTTP backends. Implement a subclass to talk to a backend over HTTP which is not supported by the platform out of the box.

Implementations must override one of the unmarshal methods to unmarshal the response.

Author:

Arne Bergene Fossaa, bratseth

Nested Class Summary

Nested classes/interfaces inherited from class com.yahoo.search.federation.http.HTTPSearcher

HTTPSearcher.SearcherHttpClient, HTTPSearcher.ThrowingCertificateStore

Field Summary

Fields

Modifier and Type	Field and Description
protected String	cacheName The name of the cache used (which is just getid().stringValue(), or null if no cache is used
private com.yahoo.statistics.Counter	emptyResults
private static List<String>	excludedSourceProperties
private com.yahoo.statistics.Value	hitsPerQuery
protected static Logger	log
private com.yahoo.statistics.Counter	readTimeouts
private com.yahoo.statistics.Value	responseLatency

Fields inherited from class com.yahoo.search.federation.http.HTTPSearcher

certificateProxy, IP_ADDRESS_UNKNOWN, LOG_CONNECT_TIMEOUT_PREFIX, LOG_HEADER_PREFIX, LOG_HITCOUNT, LOG_HOST, LOG_IP_ADDRESS, LOG_LATENCY_CONNECT, LOG_LATENCY_FINISH, LOG_LATENCY_START, LOG_PATH, LOG_PORT, LOG_PROXY_HOST, LOG_PROXY_PORT, LOG_PROXY_TYPE, LOG_QUERY_PARAM_PREFIX, LOG_READ_TIMEOUT_PREFIX, LOG_RESPONSE_HEADER_PREFIX, LOG_SCHEME, LOG_STATUS, LOG_URI, useCertificate, YCA_HTTP_HEADER

Fields inherited from class com.yahoo.component.AbstractComponent

isDeconstructable

Constructor Summary

Constructors

Constructor and Description
HTTPProviderSearcher(com.yahoo.component.ComponentId id, List<Connection> connections, HTTPParameters parameters, com.yahoo.statistics.Statistics statistics)
HTTPProviderSearcher(com.yahoo.component.ComponentId id, List<Connection> connections, HTTPParameters parameters, com.yahoo.statistics.Statistics statistics, com.yahoo.jdisc.http.CertificateStore certificateStore) Creates a provider searcher
HTTPProviderSearcher(com.yahoo.component.ComponentId id, List<Connection> connections, String path, com.yahoo.statistics.Statistics statistics)
HTTPProviderSearcher(com.yahoo.component.ComponentId id, List<Connection> connections, String path, com.yahoo.statistics.Statistics statistics, com.yahoo.jdisc.http.CertificateStore certificateStore) Creates a http provider searcher using id.getName as provider name

Method Summary

Modifier and Type	Method and Description
protected void	addNonExcludedSourceProperties(Query query, Map<String,String> queryMap)
protected void	configureCache(QrBinaryCacheConfig cacheConfig, QrBinaryCacheRegionConfig regionConfig) Deprecated. this method does nothing
private Result	consumeEntity(org.apache.http.HttpEntity entity, Query query, Result result, Hit logHit, long requestStartTime)
abstract Map<String,String>	getCacheKey(Query q) Deprecated. the cache key is ignored as there is no built-in caching support
Map<String,String>	getQueryMap(Query query) Returns the key-value pairs that should be added as properties to the request url sent to the service.
protected Result	inspectAndReturnFinalResult(Result result, Hit requestMeta, Exception e) Hook called at the moment the result is returned from this searcher.
protected void	logResponseLatency(long latency)
Result	search(Query query, Execution execution, Connection connection) Perform the search against the given connection.
private Result	statisticsBeforeInspection(Result result, Hit requestMeta, Exception e)
void	unmarshal(org.apache.http.HttpEntity entity, Result result) Unmarshal the result from an http entity.
void	unmarshal(InputStream stream, long contentLength, Result result) Unmarshal the stream by converting it to hits and adding the hits to the given result.

Methods inherited from class com.yahoo.search.federation.http.HTTPSearcher

checkPing, cleanupHttpEntity, createClient, createMalformedUrlError, createRequest, createRequest, createRequestMeta, getConnectionManager, getEntity, getEntity, getParameters, getPath, getPingPath, getPingURI, getRequestEntity, getRequestHeaders, getResponse, getResponse, getResponse, getResponse, getURI, getURI, ping, shutdownConnectionManagers

Methods inherited from class com.yahoo.search.cluster.ClusterSearcher

deconstruct, failed, fill, fill, getFirstConnection, getHasher, getMonitor, log, ping, robustSearch, search, search, shouldRetry, timedOut, working

Methods inherited from class com.yahoo.search.cluster.PingableSearcher

ping

Methods inherited from class com.yahoo.search.Searcher

ensureFilled, getLogger, process, toString

Methods inherited from class com.yahoo.component.chain.ChainedComponent

getAnnotatedDependencies, getDefaultAnnotatedDependencies, getDependencies, initDependencies

Methods inherited from class com.yahoo.component.AbstractComponent

clone, compareTo, getClassName, getId, getIdString, hasInitializedId, initId, isDeconstructable, setIsDeconstructable

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.yahoo.search.cluster.NodeManager

pingIterationCompleted

Field Detail

emptyResults

private final com.yahoo.statistics.Counter emptyResults

hitsPerQuery

private final com.yahoo.statistics.Value hitsPerQuery

responseLatency

private final com.yahoo.statistics.Value responseLatency

readTimeouts

private final com.yahoo.statistics.Counter readTimeouts

excludedSourceProperties

private static final List<String> excludedSourceProperties

log

protected static final Logger log

cacheName

protected String cacheName

The name of the cache used (which is just getid().stringValue(), or null if no cache is used

Constructor Detail

HTTPProviderSearcher

public HTTPProviderSearcher(com.yahoo.component.ComponentId id,
                            List<Connection> connections,
                            String path,
                            com.yahoo.statistics.Statistics statistics)

HTTPProviderSearcher

public HTTPProviderSearcher(com.yahoo.component.ComponentId id,
                            List<Connection> connections,
                            String path,
                            com.yahoo.statistics.Statistics statistics,
                            com.yahoo.jdisc.http.CertificateStore certificateStore)

Creates a http provider searcher using id.getName as provider name

HTTPProviderSearcher

public HTTPProviderSearcher(com.yahoo.component.ComponentId id,
                            List<Connection> connections,
                            HTTPParameters parameters,
                            com.yahoo.statistics.Statistics statistics)

HTTPProviderSearcher

public HTTPProviderSearcher(com.yahoo.component.ComponentId id,
                            List<Connection> connections,
                            HTTPParameters parameters,
                            com.yahoo.statistics.Statistics statistics,
                            com.yahoo.jdisc.http.CertificateStore certificateStore)

Creates a provider searcher

Parameters:

id - the id of this instance

connections - the connections this will load balance and fail over between

parameters - the parameters to use when making http calls

Method Detail

configureCache

@Deprecated
protected void configureCache(QrBinaryCacheConfig cacheConfig,
                                          QrBinaryCacheRegionConfig regionConfig)

Deprecated. this method does nothing

unmarshal

public void unmarshal(InputStream stream,
                      long contentLength,
                      Result result)
               throws IOException

Unmarshal the stream by converting it to hits and adding the hits to the given result. A convenience hook called by the default unmarshal(entity,result). Override this in subclasses which does not override unmarshal(entity,result).

This default implementation throws an exception.

Parameters:

stream - the stream of data returned

contentLength - the length of the content in bytes if known, or a negative number if unknown

result - the result to which unmarshalled data should be added

Throws:

IOException

unmarshal

public void unmarshal(org.apache.http.HttpEntity entity,
                      Result result)
               throws IOException

Unmarshal the result from an http entity. This default implementation calls unmarshal(entity.getContent(), entity.getContentLength(), result) (and does some detailed query tracing).

Parameters:

entity - the entity containing the data to unmarshal

result - the result to which unmarshalled data should be added

Throws:

IOException

addNonExcludedSourceProperties

protected void addNonExcludedSourceProperties(Query query,
                                              Map<String,String> queryMap)

inspectAndReturnFinalResult

protected Result inspectAndReturnFinalResult(Result result,
                                             Hit requestMeta,
                                             Exception e)

Hook called at the moment the result is returned from this searcher. This default implementation does return result.

Parameters:

result - the result which is to be returned

requestMeta - the request information hit, or null if none was created (e.g if this was a cache lookup)

e - the exception caused during execution of this query, or null if none

Returns:

the result which is returned upwards

statisticsBeforeInspection

private Result statisticsBeforeInspection(Result result,
                                          Hit requestMeta,
                                          Exception e)

logResponseLatency

protected void logResponseLatency(long latency)

Overrides:

logResponseLatency in class HTTPSearcher

search

public Result search(Query query,
                     Execution execution,
                     Connection connection)

Description copied from class: ClusterSearcher

Perform the search against the given connection. Return a result containing an error or throw an exception on failures.

Specified by:

search in class ClusterSearcher<Connection>

consumeEntity

private Result consumeEntity(org.apache.http.HttpEntity entity,
                             Query query,
                             Result result,
                             Hit logHit,
                             long requestStartTime)

getQueryMap

public Map<String,String> getQueryMap(Query query)

Returns the key-value pairs that should be added as properties to the request url sent to the service. Must be overridden in subclasses to add the key-values expected by the service in question, unless HTTPSearcher.getURI(com.yahoo.search.Query, com.yahoo.search.federation.http.Connection) (from which this is called) is overridden.

This default implementation returns the query.properties() prefixed by "source.[sourceName]" or "property.[propertyName]" (by calling addNonExcludedSourceProperties(com.yahoo.search.Query, java.util.Map<java.lang.String, java.lang.String>)).

Overrides:

getQueryMap in class HTTPSearcher

getCacheKey

@Deprecated
public abstract Map<String,String> getCacheKey(Query q)

Deprecated. the cache key is ignored as there is no built-in caching support