org.archive.modules.deciderules.surt

Class SurtPrefixedDecideRule

java.lang.Object

org.archive.modules.deciderules.DecideRule

org.archive.modules.deciderules.PredicatedDecideRule

org.archive.modules.deciderules.surt.SurtPrefixedDecideRule

All Implemented Interfaces:

Serializable, EventListener, Checkpointable, SeedListener, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>

Direct Known Subclasses:

NotSurtPrefixedDecideRule, OnDomainsDecideRule, OnHostsDecideRule

public class SurtPrefixedDecideRule
extends PredicatedDecideRule
implements SeedListener, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>, Checkpointable, org.springframework.beans.factory.BeanNameAware

Rule applies configured decision to any URIs that, when expressed in SURT form, begin with one of the prefixes in the configured set. The set can be filled with SURT prefixes implied or listed in the seeds file, or another external file. The "also-check-via" option to implement "one hop off" scoping derives from a contribution by Shifra Raffel of the California Digital Library.

Author:

gojomo

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String	beanName
protected Checkpoint	recoveryCheckpoint
protected SeedModule	seeds
protected boolean	seedsAsSurtPrefixes Should seeds also be interpreted as SURT prefixes.
protected SurtPrefixSet	surtPrefixes
protected ConfigFile	surtsDumpFile Dump file to save SURT prefixes actually used: Useful debugging SURTs.
protected ReadSource	surtsSource Text from which to infer SURT prefixes.

Fields inherited from class org.archive.modules.deciderules.DecideRule

comment, kp

Constructor Summary

Constructors

Constructor and Description
SurtPrefixedDecideRule()

Method Summary

Modifier and Type	Method and Description
void	addedSeed(CrawlURI curi) If appropriate, convert seed notification into prefix-addition.
protected void	buildSurtPrefixSet() Construct the set of prefixes to use, from the seed list ( which may include both URIs and '+'-prefixed directives).
void	concludedSeedBatch()
void	doCheckpoint(Checkpoint checkpointInProgress)
protected void	dumpSurtPrefixSet() Dump the current prefixes in use to configured dump file (if any)
protected boolean	evaluate(CrawlURI uri) Evaluate whether given object's URI is covered by the SURT prefix set
void	finishCheckpoint(Checkpoint checkpointInProgress)
boolean	getAlsoCheckVia()
SeedModule	getSeeds()
boolean	getSeedsAsSurtPrefixes()
ConfigFile	getSurtsDumpFile()
ReadSource	getSurtsSource()
ConfigFile	getSurtsSourceFile() Deprecated. redundant now that we have surtsSource
boolean	nonseedLine(String line) Consider nonseed lines as possible SURT prefix directives.
void	onApplicationEvent(org.springframework.context.ApplicationEvent event)
protected String	prefixFrom(String uri)
protected void	readPrefixes()
void	setAlsoCheckVia(boolean checkVia)
void	setBeanName(String name)
void	setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)
void	setSeeds(SeedModule seeds)
void	setSeedsAsSurtPrefixes(boolean seedsAsSurtPrefixes)
void	setSurtsDumpFile(ConfigFile cp)
void	setSurtsSource(ReadSource surtsSource)
void	setSurtsSourceFile(ConfigFile cp) Deprecated.
void	startCheckpoint(Checkpoint checkpointInProgress)

Methods inherited from class org.archive.modules.deciderules.PredicatedDecideRule

getDecision, innerDecide, onlyDecision, setDecision

Methods inherited from class org.archive.modules.deciderules.DecideRule

accepts, decisionFor, getComment, getEnabled, getKeyedProperties, setComment, setEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

surtsSource

protected ReadSource surtsSource

Text from which to infer SURT prefixes. Any URLs will be converted to the implied SURT prefix, and literal SURT prefixes may be listed on lines beginning with a '+' character.

seedsAsSurtPrefixes

protected boolean seedsAsSurtPrefixes

Should seeds also be interpreted as SURT prefixes.

surtsDumpFile

protected ConfigFile surtsDumpFile

Dump file to save SURT prefixes actually used: Useful debugging SURTs.

seeds

protected SeedModule seeds

surtPrefixes

protected SurtPrefixSet surtPrefixes

beanName

protected String beanName

recoveryCheckpoint

protected Checkpoint recoveryCheckpoint

Constructor Detail

SurtPrefixedDecideRule

public SurtPrefixedDecideRule()

Method Detail

getSurtsSourceFile

public ConfigFile getSurtsSourceFile()

Deprecated. redundant now that we have surtsSource

Source file from which to infer SURT prefixes. Any URLs in file will be converted to the implied SURT prefix, and literal SURT prefixes may be listed on lines beginning with a '+' character.

setSurtsSourceFile

public void setSurtsSourceFile(ConfigFile cp)

Deprecated.

getSurtsSource

public ReadSource getSurtsSource()

setSurtsSource

public void setSurtsSource(ReadSource surtsSource)

getSeedsAsSurtPrefixes

public boolean getSeedsAsSurtPrefixes()

setSeedsAsSurtPrefixes

public void setSeedsAsSurtPrefixes(boolean seedsAsSurtPrefixes)

getSurtsDumpFile

public ConfigFile getSurtsDumpFile()

setSurtsDumpFile

public void setSurtsDumpFile(ConfigFile cp)

getAlsoCheckVia

public boolean getAlsoCheckVia()

setAlsoCheckVia

public void setAlsoCheckVia(boolean checkVia)

getSeeds

public SeedModule getSeeds()

setSeeds

public void setSeeds(SeedModule seeds)

concludedSeedBatch

public void concludedSeedBatch()

Specified by:

concludedSeedBatch in interface SeedListener

evaluate

protected boolean evaluate(CrawlURI uri)

Evaluate whether given object's URI is covered by the SURT prefix set

Specified by:

evaluate in class PredicatedDecideRule

Returns:

true if item, as SURT form URI, is prefixed by an item in the set

readPrefixes

protected void readPrefixes()

dumpSurtPrefixSet

protected void dumpSurtPrefixSet()

Dump the current prefixes in use to configured dump file (if any)

buildSurtPrefixSet

protected void buildSurtPrefixSet()

Construct the set of prefixes to use, from the seed list ( which may include both URIs and '+'-prefixed directives).

addedSeed

public void addedSeed(CrawlURI curi)

If appropriate, convert seed notification into prefix-addition.

Specified by:

addedSeed in interface SeedListener

See Also:

SeedListener.addedSeed(org.archive.modules.CrawlURI)

prefixFrom

protected String prefixFrom(String uri)

nonseedLine

public boolean nonseedLine(String line)

Consider nonseed lines as possible SURT prefix directives. If we are ACCEPTing URIs, expect '+' as the add-SURT directive. If we are REJECTing URIs, expect '-' as the add-SURT directive. TODO: perhaps, make directive configurable, so many SeedListener SurtPrefixedDecideRules can all listen for directives meant only for them.

Specified by:

nonseedLine in interface SeedListener

See Also:

SeedListener.nonseedLine(java.lang.String)

onApplicationEvent

public void onApplicationEvent(org.springframework.context.ApplicationEvent event)

Specified by:

onApplicationEvent in interface org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>

setBeanName

public void setBeanName(String name)

Specified by:

setBeanName in interface org.springframework.beans.factory.BeanNameAware

startCheckpoint

public void startCheckpoint(Checkpoint checkpointInProgress)

Specified by:

startCheckpoint in interface Checkpointable

doCheckpoint

public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException

Specified by:

doCheckpoint in interface Checkpointable

Throws:

IOException

finishCheckpoint

public void finishCheckpoint(Checkpoint checkpointInProgress)

Specified by:

finishCheckpoint in interface Checkpointable

setRecoveryCheckpoint

public void setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)

Specified by:

setRecoveryCheckpoint in interface Checkpointable