org.archive.modules.writer

Class BaseWARCWriterProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.BaseWARCWriterProcessor

All Implemented Interfaces:

Checkpointable, org.archive.io.warc.WARCWriterPoolSettings, WriterPoolSettings, HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Direct Known Subclasses:

WARCWriterChainProcessor, WARCWriterProcessor

public abstract class BaseWARCWriterProcessor
extends WriterPoolProcessor
implements org.archive.io.warc.WARCWriterPoolSettings

Field Summary

Fields

Modifier and Type	Field and Description
protected org.archive.uid.RecordIDGenerator	generator Generator for record IDs
protected ConcurrentMap<String,ConcurrentMap<String,AtomicLong>>	stats
protected AtomicLong	urlsWritten

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
BaseWARCWriterProcessor()

Method Summary

Modifier and Type	Method and Description
protected void	addIfNotBlank(org.archive.util.anvl.ANVLRecord record, String label, String value)
protected void	addStats(Map<String,Map<String,Long>> substats)
protected Map<String,Map<String,Long>>	copyStats(Map<String,Map<String,Long>> orig)
long	getDefaultMaxFileSize()
List<ConfigPath>	getDefaultStorePaths()
List<String>	getMetadata()
protected URI	getRecordID()
org.archive.uid.RecordIDGenerator	getRecordIDGenerator()
ConcurrentMap<String,ConcurrentMap<String,AtomicLong>>	getStats()
String	report()
void	setRecordIDGenerator(org.archive.uid.RecordIDGenerator generator)
protected void	setupPool(AtomicInteger serialNo) Set up pool of files.
protected void	updateMetadataAfterWrite(CrawlURI curi, org.archive.io.warc.WARCWriter writer, long startPosition)

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

addTotalBytesWritten, calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, fromCheckpointJson, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerProcessResult, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, shouldProcess, shouldWrite, start, stop, toCheckpointJson

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.io.WriterPoolSettings

calcOutputDirs, getCompress, getFrequentFlushes, getMaxFileSizeBytes, getPrefix, getTemplate, getWriteBufferSize

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Field Detail

urlsWritten

protected AtomicLong urlsWritten

stats

protected ConcurrentMap<String,ConcurrentMap<String,AtomicLong>> stats

generator

protected org.archive.uid.RecordIDGenerator generator

Generator for record IDs

Constructor Detail

BaseWARCWriterProcessor

public BaseWARCWriterProcessor()

Method Detail

getStats

public ConcurrentMap<String,ConcurrentMap<String,AtomicLong>> getStats()

getRecordIDGenerator

public org.archive.uid.RecordIDGenerator getRecordIDGenerator()

Specified by:

getRecordIDGenerator in interface org.archive.io.warc.WARCWriterPoolSettings

setRecordIDGenerator

public void setRecordIDGenerator(org.archive.uid.RecordIDGenerator generator)

getRecordID

protected URI getRecordID()
                   throws IOException

Throws:

IOException

getDefaultMaxFileSize

public long getDefaultMaxFileSize()

Specified by:

getDefaultMaxFileSize in class WriterPoolProcessor

getDefaultStorePaths

public List<ConfigPath> getDefaultStorePaths()

Specified by:

getDefaultStorePaths in class WriterPoolProcessor

setupPool

protected void setupPool(AtomicInteger serialNo)

Description copied from class: WriterPoolProcessor

Set up pool of files.

Specified by:

setupPool in class WriterPoolProcessor

getMetadata

public List<String> getMetadata()

Specified by:

getMetadata in interface WriterPoolSettings

Specified by:

getMetadata in class WriterPoolProcessor

addIfNotBlank

protected void addIfNotBlank(org.archive.util.anvl.ANVLRecord record,
                             String label,
                             String value)

addStats

protected void addStats(Map<String,Map<String,Long>> substats)

report

public String report()

Overrides:

report in class Processor

copyStats

protected Map<String,Map<String,Long>> copyStats(Map<String,Map<String,Long>> orig)

updateMetadataAfterWrite

protected void updateMetadataAfterWrite(CrawlURI curi,
                                        org.archive.io.warc.WARCWriter writer,
                                        long startPosition)