Package org.archive.modules.postprocessor

Class AMQPCrawlLogFeed

java.lang.Object

org.archive.modules.Processor

org.archive.modules.AMQPProducerProcessor

org.archive.modules.postprocessor.AMQPCrawlLogFeed

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class AMQPCrawlLogFeed
extends AMQPProducerProcessor
implements org.springframework.context.Lifecycle

Author:

nlevitt

See Also:

• UriProcessingFormatter

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	dumpPendingAtClose
protected Map<String,String>	extraFields
protected Frontier	frontier
protected com.rabbitmq.client.AMQP.BasicProperties	props
protected ServerCache	serverCache

Fields inherited from class org.archive.modules.AMQPProducerProcessor

amqpProducer, logger

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor	Description
AMQPCrawlLogFeed()

Method Summary

Modifier and Type	Method	Description
protected com.rabbitmq.client.AMQP.BasicProperties	amqpMessageProperties()
protected byte[]	buildMessage(CrawlURI curi)
boolean	getDumpPendingAtClose()
Map<String,String>	getExtraFields()
Frontier	getFrontier()
ServerCache	getServerCache()
void	setDumpPendingAtClose(boolean dumpPendingAtClose)	If true, publish all pending urls (i.e. queued urls still in the frontier) when crawl job is stopping.
void	setExtraFields(Map<String,String> extraFields)
void	setFrontier(Frontier frontier)	Autowired frontier, needed to determine when a url is finished.
void	setServerCache(ServerCache serverCache)
protected boolean	shouldProcess(CrawlURI curi)
void	stop()

Methods inherited from class org.archive.modules.AMQPProducerProcessor

amqpProducer, fail, getAmqpUri, getExchange, getRoutingKey, innerProcess, innerProcessResult, setAmqpUri, setExchange, setRoutingKey, success

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.context.Lifecycle

isRunning, start

Field Details

frontier

protected Frontier frontier

serverCache

protected ServerCache serverCache

extraFields

protected Map<String,String> extraFields

dumpPendingAtClose

protected boolean dumpPendingAtClose

props

protected com.rabbitmq.client.AMQP.BasicProperties props

Constructor Details

AMQPCrawlLogFeed

public AMQPCrawlLogFeed()

Method Details

getFrontier

public Frontier getFrontier()

setFrontier

@Autowired
public void setFrontier(Frontier frontier)

Autowired frontier, needed to determine when a url is finished.

getServerCache

public ServerCache getServerCache()

setServerCache

@Autowired
public void setServerCache(ServerCache serverCache)

getExtraFields

public Map<String,String> getExtraFields()

setExtraFields

public void setExtraFields(Map<String,String> extraFields)

getDumpPendingAtClose

public boolean getDumpPendingAtClose()

setDumpPendingAtClose

public void setDumpPendingAtClose(boolean dumpPendingAtClose)

If true, publish all pending urls (i.e. queued urls still in the frontier) when crawl job is stopping. They are recognizable by the status field which has the value 0.

See Also:

• BdbFrontier.setDumpPendingAtClose(boolean)

buildMessage

protected byte[] buildMessage(CrawlURI curi)

Specified by:

buildMessage in class AMQPProducerProcessor

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Specified by:

shouldProcess in class Processor

stop

public void stop()

Specified by:

stop in interface org.springframework.context.Lifecycle

Overrides:

stop in class AMQPProducerProcessor

amqpMessageProperties

protected com.rabbitmq.client.AMQP.BasicProperties amqpMessageProperties()

Overrides:

amqpMessageProperties in class AMQPProducerProcessor