org.archive.modules.recrawl

Class PersistLoadProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.recrawl.AbstractPersistProcessor

org.archive.modules.recrawl.PersistProcessor

org.archive.modules.recrawl.PersistOnlineProcessor

org.archive.modules.recrawl.PersistLoadProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class PersistLoadProcessor
extends PersistOnlineProcessor

Loads CrawlURI attributes from previous fetch from persistent storage for consultation by a later recrawl.

Version:

$Date: 2006-09-25 20:19:54 +0000 (Mon, 25 Sep 2006) $, $Revision: 4654 $

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected ConfigPath	preloadSource A source (either log file or BDB directory) from which to copy history information into the current store at startup.
protected String	preloadSourceUrl A log file source url from which to copy history information into the current store at startup.

Fields inherited from class org.archive.modules.recrawl.PersistOnlineProcessor

bdb, historyDb, historyDbName, store

Fields inherited from class org.archive.modules.recrawl.PersistProcessor

HISTORY_DB_CONFIG, URI_HISTORY_DBNAME

Fields inherited from class org.archive.modules.recrawl.AbstractPersistProcessor

onlyStoreIfWriteTagPresent

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
PersistLoadProcessor()

Method Summary

Modifier and Type	Method and Description
ConfigPath	getPreloadSource()
String	getPreloadSourceUrl()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	setPreloadSource(ConfigPath preloadSource)
void	setPreloadSourceUrl(String preloadSourceUrl)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.
void	start()

Methods inherited from class org.archive.modules.recrawl.PersistOnlineProcessor

getHistoryDbName, isRunning, setBdbModule, setHistoryDbName, stop

Methods inherited from class org.archive.modules.recrawl.PersistProcessor

copyPersistSourceToHistoryMap, copyPersistSourceToHistoryMap, main, persistKeyFor, persistKeyFor, populatePersistEnv, setupCopyEnvironment, setupCopyEnvironment

Methods inherited from class org.archive.modules.recrawl.AbstractPersistProcessor

getOnlyStoreIfWriteTagPresent, hasWriteTag, setOnlyStoreIfWriteTagPresent, shouldLoad, shouldStore

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

preloadSource

protected ConfigPath preloadSource

A source (either log file or BDB directory) from which to copy history information into the current store at startup. (Whenever possible, it would be better to ensure the original history DB is in its own independent BDB environment, and then copy and reuse that environment in the followup crawl(s).) Only one of {link @preloadSource} and {link @preloadSourceUrl} may be specified.

preloadSourceUrl

protected String preloadSourceUrl

A log file source url from which to copy history information into the current store at startup. (Whenever possible, it would be better to ensure the original history DB is in its own independent BDB environment, and then copy and reuse that environment in the followup crawl(s).) Only one of preloadSource and preloadSourceUrl may be specified.

Constructor Detail

PersistLoadProcessor

public PersistLoadProcessor()

Method Detail

getPreloadSource

public ConfigPath getPreloadSource()

setPreloadSource

public void setPreloadSource(ConfigPath preloadSource)

getPreloadSourceUrl

public String getPreloadSourceUrl()

setPreloadSourceUrl

public void setPreloadSourceUrl(String preloadSourceUrl)

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the Processor.getEnabled(), the Processor.getShouldProcessRule() and the Processor.shouldProcess(CrawlURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

Throws:

InterruptedException - if the thread is interrupted

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

start

public void start()

Specified by:

start in interface org.springframework.context.Lifecycle

Overrides:

start in class PersistOnlineProcessor