Package org.archive.crawler.prefetch

Class HostQuotaEnforcer

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.prefetch.HostQuotaEnforcer

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class HostQuotaEnforcer
extends Processor

Enforces quotas on a host. Should be configured early in FetchChain in crawler-beans.cxml. Supports quotas on any of the fields tracked in FetchStats.

Author:

nlevitt

See Also:

• QuotaEnforcer
• SourceQuotaEnforcer

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	applyToSubdomains
protected String	host
protected Map<String,Long>	quotas
protected ServerCache	serverCache

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor	Description
HostQuotaEnforcer()

Method Summary

Modifier and Type	Method	Description
boolean	getApplyToSubdomains()
String	getHost()
Map<String,Long>	getQuotas()
ServerCache	getServerCache()
protected void	innerProcess(CrawlURI curi)
protected ProcessResult	innerProcessResult(CrawlURI curi)
void	setApplyToSubdomains(boolean applyToSubdomains)	Whether to apply the quotas to each subdomain of host (separately, not cumulatively).
void	setHost(String host)
void	setQuotas(Map<String,Long> quotas)	Keys can be any of the FetchStats keys.
void	setServerCache(ServerCache serverCache)
protected boolean	shouldProcess(CrawlURI curi)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

serverCache

protected ServerCache serverCache

host

protected String host

applyToSubdomains

protected boolean applyToSubdomains

quotas

protected Map<String,Long> quotas

Constructor Details

HostQuotaEnforcer

public HostQuotaEnforcer()

Method Details

getServerCache

public ServerCache getServerCache()

setServerCache

@Autowired
public void setServerCache(ServerCache serverCache)

getHost

public String getHost()

setHost

public void setHost(String host)

getApplyToSubdomains

public boolean getApplyToSubdomains()

setApplyToSubdomains

public void setApplyToSubdomains(boolean applyToSubdomains)

Whether to apply the quotas to each subdomain of host (separately, not cumulatively).

getQuotas

public Map<String,Long> getQuotas()

setQuotas

public void setQuotas(Map<String,Long> quotas)

Keys can be any of the FetchStats keys.

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Specified by:

shouldProcess in class Processor

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Specified by:

innerProcess in class Processor

Throws:

InterruptedException

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)
                                    throws InterruptedException

Overrides:

innerProcessResult in class Processor

Throws:

InterruptedException