org.archive.modules.writer

Class WARCWriterChainProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.BaseWARCWriterProcessor

org.archive.modules.writer.WARCWriterChainProcessor

All Implemented Interfaces:

Checkpointable, org.archive.io.warc.WARCWriterPoolSettings, WriterPoolSettings, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class WARCWriterChainProcessor
extends BaseWARCWriterProcessor
implements HasKeyedProperties

WARC writer processor. The types of records that to be written can be configured by including or excluding WARCRecordBuilder implementations (see setChain(List)).

This is the default chain:

   <property name="chain">
    <list>
     <bean class="org.archive.modules.warc.DnsResponseRecordBuilder"/>
     <bean class="org.archive.modules.warc.HttpResponseRecordBuilder"/>
     <bean class="org.archive.modules.warc.WhoisResponseRecordBuilder"/>
     <bean class="org.archive.modules.warc.FtpControlConversationRecordBuilder"/>
     <bean class="org.archive.modules.warc.FtpResponseRecordBuilder"/>
     <bean class="org.archive.modules.warc.RevisitRecordBuilder"/>
     <bean class="org.archive.modules.warc.HttpRequestRecordBuilder"/>
     <bean class="org.archive.modules.warc.MetadataRecordBuilder"/>
    </list>
   </property>
 

Replaces WARCWriterProcessor.

Author:

nlevitt

See Also:

WARCRecordBuilder

Field Summary

Fields inherited from class org.archive.modules.writer.BaseWARCWriterProcessor

generator, stats, urlsWritten

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
WARCWriterChainProcessor()

Method Summary

Modifier and Type	Method and Description
List<? extends WARCRecordBuilder>	getChain()
protected ProcessResult	innerProcessResult(CrawlURI curi)
void	setChain(List<? extends WARCRecordBuilder> chain)
protected boolean	shouldWrite(CrawlURI curi) Whether the given CrawlURI should be written to archive files.
protected ProcessResult	write(CrawlURI curi)
protected void	writeRecords(CrawlURI curi, org.archive.io.warc.WARCWriter writer)

Methods inherited from class org.archive.modules.writer.BaseWARCWriterProcessor

addIfNotBlank, addStats, copyStats, getDefaultMaxFileSize, getDefaultStorePaths, getMetadata, getRecordID, getRecordIDGenerator, getStats, report, setRecordIDGenerator, setupPool, updateMetadataAfterWrite

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, fromCheckpointJson, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, shouldProcess, start, stop, toCheckpointJson

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.spring.HasKeyedProperties

getKeyedProperties

Methods inherited from interface org.archive.io.WriterPoolSettings

calcOutputDirs, getCompress, getFrequentFlushes, getMaxFileSizeBytes, getPrefix, getTemplate, getWriteBufferSize

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Constructor Detail

WARCWriterChainProcessor

public WARCWriterChainProcessor()

Method Detail

getChain

public List<? extends WARCRecordBuilder> getChain()

setChain

public void setChain(List<? extends WARCRecordBuilder> chain)

shouldWrite

protected boolean shouldWrite(CrawlURI curi)

Description copied from class: WriterPoolProcessor

Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer.

Overrides:

shouldWrite in class WriterPoolProcessor

Parameters:

curi - CrawlURI

Returns:

true if URI should be written; false otherwise

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)

Specified by:

innerProcessResult in class WriterPoolProcessor

write

protected ProcessResult write(CrawlURI curi)
                       throws IOException

Throws:

IOException

writeRecords

protected void writeRecords(CrawlURI curi,
                            org.archive.io.warc.WARCWriter writer)
                     throws IOException

Throws:

IOException