WriterPoolProcessor (Heritrix 3: 'modules' subproject (reusable components) 3.4.0-20190418 API)

java.lang.Object
- org.archive.modules.Processor
- - org.archive.modules.writer.WriterPoolProcessor

All Implemented Interfaces:

Checkpointable, WriterPoolSettings, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Direct Known Subclasses:

ARCWriterProcessor, WARCWriterProcessor
```
public abstract class WriterPoolProcessor
extends Processor
implements org.springframework.context.Lifecycle, Checkpointable, WriterPoolSettings
```
Abstract implementation of a file pool processor. Subclass to implement for a particular WriterPoolMember instance.

Author:

Parker Thompson, stack

Field Summary

Fields
Modifier and Type	Field and Description
`protected static String`	`ANNOTATION_UNWRITTEN` CrawlURI annotation indicating no record was written.
`protected boolean`	`compress` Whether to gzip-compress files when writing to disk; by default true, meaning do-compress.
`protected ConfigPath`	`directory`
`protected boolean`	`frequentFlushes` Whether to flush to underlying file frequently (at least after each record), or not.
`protected long`	`maxFileSizeBytes` Max size of each file.
`protected long`	`maxTotalBytesToWrite` Total file bytes to write to disk.
`protected int`	`maxWaitForIdleMs` Maximum time to wait on idle writer before (possibly) creating an additional instance.
`protected int`	`poolMaxActive` Maximum active files in pool.
`protected String`	`prefix` File prefix.
`protected ServerCache`	`serverCache`
`protected boolean`	`skipIdenticalDigests` Whether to skip the writing of a record when URI history information is available and indicates the prior fetch had an identical content digest.
`protected boolean`	`startNewFilesOnCheckpoint`
`protected List<ConfigPath>`	`storePaths` Where to save files.
`protected String`	`template` Template from which a filename is interpolated.
`protected int`	`writeBufferSize` Size of buffer in front of disk-writing.

Fields inherited from class org.archive.modules.Processor
beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors
Constructor and Description

WriterPoolProcessor()

Constructors
Constructor and Description
`WriterPoolProcessor()`

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`List<File>`	`calcOutputDirs()`
`protected ProcessResult`	`checkBytesWritten()`
`protected void`	`copyForwardWriteTagIfDupe(CrawlURI curi)` If this fetch is identical to the last written (archived) fetch, then copy forward the writeTag.
`void`	`doCheckpoint(Checkpoint checkpointInProgress)`
`protected void`	`fromCheckpointJson(org.json.JSONObject json)` Restore internal state from JSONObject stored at earlier checkpoint-time.
`boolean`	`getCompress()`
`protected abstract long`	`getDefaultMaxFileSize()`
`protected abstract List<ConfigPath>`	`getDefaultStorePaths()`
`ConfigPath`	`getDirectory()`
`boolean`	`getFrequentFlushes()`
`protected String`	`getHostAddress(CrawlURI curi)` Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line).
`long`	`getMaxFileSizeBytes()`
`long`	`getMaxTotalBytesToWrite()`
`int`	`getMaxWaitForIdleMs()`
`abstract List<String>`	`getMetadata()`
`CrawlMetadata`	`getMetadataProvider()`
`protected WriterPool`	`getPool()`
`int`	`getPoolMaxActive()`
`String`	`getPrefix()`
`protected AtomicInteger`	`getSerialNo()`
`ServerCache`	`getServerCache()`
`boolean`	`getSkipIdenticalDigests()`
`boolean`	`getStartNewFilesOnCheckpoint()`
`List<ConfigPath>`	`getStorePaths()`
`String`	`getTemplate()`
`protected long`	`getTotalBytesWritten()`
`int`	`getWriteBufferSize()`
`protected void`	`innerProcess(CrawlURI puri)` Actually performs the process.
`protected abstract ProcessResult`	`innerProcessResult(CrawlURI uri)`
`protected void`	`innerRejectProcess(CrawlURI curi)` Invoked after a URI has been rejected.
`void`	`setCompress(boolean compress)`
`void`	`setDirectory(ConfigPath directory)`
`void`	`setFrequentFlushes(boolean frequentFlushes)`
`void`	`setMaxFileSizeBytes(long maxFileSizeBytes)`
`void`	`setMaxTotalBytesToWrite(long maxTotalBytesToWrite)`
`void`	`setMaxWaitForIdleMs(int maxWaitForIdle)`
`void`	`setMetadataProvider(CrawlMetadata provider)`
`protected void`	`setPool(WriterPool pool)`
`void`	`setPoolMaxActive(int poolMaxActive)`
`void`	`setPrefix(String prefix)`
`void`	`setServerCache(ServerCache serverCache)`
`void`	`setSkipIdenticalDigests(boolean skipIdenticalDigests)`
`void`	`setStartNewFilesOnCheckpoint(boolean startNewFilesOnCheckpoint)` Whether to close output files and start new ones on checkpoint.
`void`	`setStorePaths(List<ConfigPath> paths)`
`void`	`setTemplate(String template)`
`protected void`	`setTotalBytesWritten(long totalBytesWritten)`
`protected abstract void`	`setupPool(AtomicInteger serial)` Set up pool of files.
`void`	`setWriteBufferSize(int writeBufferSize)`
`protected boolean`	`shouldProcess(CrawlURI curi)` Determines whether the given uri should be processed by this processor.
`protected boolean`	`shouldWrite(CrawlURI curi)` Whether the given CrawlURI should be written to archive files.
`void`	`start()`
`void`	`stop()`
`protected org.json.JSONObject`	`toCheckpointJson()` Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.

Methods inherited from class org.archive.modules.Processor
finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.context.Lifecycle
isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable
finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

- Field Detail
  - compress
```
protected boolean compress
```
    Whether to gzip-compress files when writing to disk; by default true, meaning do-compress.
  - prefix
```
protected String prefix
```
    File prefix. The text supplied here will be supplied to the naming template (below) as the 'prefix' variable for possible interpolation. In the default/recommended naming formula, the prefix will appear first.
  - template
```
protected String template
```
    Template from which a filename is interpolated. Expressions of the form ${key} will be replaced by values from a local map of useful values (including 'prefix', 'timestamp17', and 'serialno') or global system properties (which includes the local hostname/port/pid). The default template is: "${prefix}-${timestamp17}-${serialno}-${heritrix.pid}~${heritrix.hostname}~${heritrix.port}" The default template will generate unique names under reasonable assumptions; be sure you know what you're doing before customizing, as you could easily create filename collisions with a poorly-designed filename template, and many downstream tools have historically assumed that ARCs/WARCs are carefully named to preserve uniqueness.
  - maxFileSizeBytes
```
protected long maxFileSizeBytes
```
    Max size of each file.
  - poolMaxActive
```
protected int poolMaxActive
```
    Maximum active files in pool. This setting cannot be varied over the life of a crawl.
  - maxWaitForIdleMs
```
protected int maxWaitForIdleMs
```
    Maximum time to wait on idle writer before (possibly) creating an additional instance.
  - skipIdenticalDigests
```
protected boolean skipIdenticalDigests
```
    Whether to skip the writing of a record when URI history information is available and indicates the prior fetch had an identical content digest. Note that subclass settings may provide more fine-grained control on how identical digest content is handled; for those controls to have effect, this setting must not be 'true' (causing content to be skipped entirely). Default is false.
  - ANNOTATION_UNWRITTEN
```
protected static final String ANNOTATION_UNWRITTEN
```
    CrawlURI annotation indicating no record was written.
    
    See Also:
    
    Constant Field Values
  - maxTotalBytesToWrite
```
protected long maxTotalBytesToWrite
```
    Total file bytes to write to disk. Once the size of all files on disk has exceeded this limit, this processor will stop the crawler. A value of zero means no upper limit.
  - frequentFlushes
```
protected boolean frequentFlushes
```
    Whether to flush to underlying file frequently (at least after each record), or not. Default is true.
  - writeBufferSize
```
protected int writeBufferSize
```
    Size of buffer in front of disk-writing. Default is 256K.
  - serverCache
```
protected transient ServerCache serverCache
```
  - directory
```
protected ConfigPath directory
```
  - startNewFilesOnCheckpoint
```
protected boolean startNewFilesOnCheckpoint
```
  - storePaths
```
protected List<ConfigPath> storePaths
```
    Where to save files. Supply absolute or relative directory paths. If relative, paths will be interpreted relative to the local 'directory' property. order.disk-path setting. If more than one path specified, we'll round-robin dropping files to each. This setting is safe to change midcrawl (You can remove and add new dirs as the crawler progresses).
- Constructor Detail
  - WriterPoolProcessor
```
public WriterPoolProcessor()
```
- Method Detail
  - getCompress
```
public boolean getCompress()
```
    Specified by:
    
    getCompress in interface WriterPoolSettings
  - setCompress
```
public void setCompress(boolean compress)
```
  - getPrefix
```
public String getPrefix()
```
    Specified by:
    
    getPrefix in interface WriterPoolSettings
  - setPrefix
```
public void setPrefix(String prefix)
```
  - getTemplate
```
public String getTemplate()
```
    Specified by:
    
    getTemplate in interface WriterPoolSettings
  - setTemplate
```
public void setTemplate(String template)
```
  - getDefaultMaxFileSize
```
protected abstract long getDefaultMaxFileSize()
```
  - getMaxFileSizeBytes
```
public long getMaxFileSizeBytes()
```
    Specified by:
    
    getMaxFileSizeBytes in interface WriterPoolSettings
  - setMaxFileSizeBytes
```
public void setMaxFileSizeBytes(long maxFileSizeBytes)
```
  - getPoolMaxActive
```
public int getPoolMaxActive()
```
  - setPoolMaxActive
```
public void setPoolMaxActive(int poolMaxActive)
```
  - getMaxWaitForIdleMs
```
public int getMaxWaitForIdleMs()
```
  - setMaxWaitForIdleMs
```
public void setMaxWaitForIdleMs(int maxWaitForIdle)
```
  - getSkipIdenticalDigests
```
public boolean getSkipIdenticalDigests()
```
  - setSkipIdenticalDigests
```
public void setSkipIdenticalDigests(boolean skipIdenticalDigests)
```
  - getMaxTotalBytesToWrite
```
public long getMaxTotalBytesToWrite()
```
  - setMaxTotalBytesToWrite
```
public void setMaxTotalBytesToWrite(long maxTotalBytesToWrite)
```
  - getFrequentFlushes
```
public boolean getFrequentFlushes()
```
    Specified by:
    
    getFrequentFlushes in interface WriterPoolSettings
  - setFrequentFlushes
```
public void setFrequentFlushes(boolean frequentFlushes)
```
  - getWriteBufferSize
```
public int getWriteBufferSize()
```
    Specified by:
    
    getWriteBufferSize in interface WriterPoolSettings
  - setWriteBufferSize
```
public void setWriteBufferSize(int writeBufferSize)
```
  - getMetadataProvider
```
public CrawlMetadata getMetadataProvider()
```
  - setMetadataProvider
```
public void setMetadataProvider(CrawlMetadata provider)
```
  - getServerCache
```
public ServerCache getServerCache()
```
  - setServerCache
```
public void setServerCache(ServerCache serverCache)
```
  - getDirectory
```
public ConfigPath getDirectory()
```
  - setDirectory
```
public void setDirectory(ConfigPath directory)
```
  - getStartNewFilesOnCheckpoint
```
public boolean getStartNewFilesOnCheckpoint()
```
  - setStartNewFilesOnCheckpoint
```
public void setStartNewFilesOnCheckpoint(boolean startNewFilesOnCheckpoint)
```
    Whether to close output files and start new ones on checkpoint. True by default. If false, merely flushes writers.
  - getDefaultStorePaths
```
protected abstract List<ConfigPath> getDefaultStorePaths()
```
  - getStorePaths
```
public List<ConfigPath> getStorePaths()
```
  - setStorePaths
```
public void setStorePaths(List<ConfigPath> paths)
```
  - start
```
public void start()
```
    Specified by:
    
    start in interface org.springframework.context.Lifecycle
    
    Overrides:
    
    start in class Processor
  - stop
```
public void stop()
```
    Specified by:
    
    stop in interface org.springframework.context.Lifecycle
    
    Overrides:
    
    stop in class Processor
  - getSerialNo
```
protected AtomicInteger getSerialNo()
```
  - setupPool
```
protected abstract void setupPool(AtomicInteger serial)
```
    Set up pool of files.
  - checkBytesWritten
```
protected ProcessResult checkBytesWritten()
```
  - shouldWrite
```
protected boolean shouldWrite(CrawlURI curi)
```
    Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer.
    
    Parameters:
    
    curi - CrawlURI
    
    Returns:
    
    true if URI should be written; false otherwise
  - getHostAddress
```
protected String getHostAddress(CrawlURI curi)
```
    Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line).
    
    Parameters:
    
    curi - CrawlURI
    
    Returns:
    
    String of IP address
  - doCheckpoint
```
public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException
```
    Specified by:
    
    doCheckpoint in interface Checkpointable
    
    Overrides:
    
    doCheckpoint in class Processor
    
    Throws:
    
    IOException
  - toCheckpointJson
```
protected org.json.JSONObject toCheckpointJson()
                                        throws org.json.JSONException
```
    Description copied from class: Processor
    
    Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.
    
    Overrides:
    
    toCheckpointJson in class Processor
    
    Returns:
    
    JSONObject
    
    Throws:
    
    org.json.JSONException
  - fromCheckpointJson
```
protected void fromCheckpointJson(org.json.JSONObject json)
                           throws org.json.JSONException
```
    Description copied from class: Processor
    
    Restore internal state from JSONObject stored at earlier checkpoint-time.
    
    Overrides:
    
    fromCheckpointJson in class Processor
    
    Parameters:
    
    json - JSONObject
    
    Throws:
    
    org.json.JSONException
  - getPool
```
protected WriterPool getPool()
```
  - setPool
```
protected void setPool(WriterPool pool)
```
  - getTotalBytesWritten
```
protected long getTotalBytesWritten()
```
  - setTotalBytesWritten
```
protected void setTotalBytesWritten(long totalBytesWritten)
```
  - getMetadata
```
public abstract List<String> getMetadata()
```
    Specified by:
    
    getMetadata in interface WriterPoolSettings
  - calcOutputDirs
```
public List<File> calcOutputDirs()
```
    Specified by:
    
    calcOutputDirs in interface WriterPoolSettings
  - innerProcess
```
protected void innerProcess(CrawlURI puri)
```
    Description copied from class: Processor
    
    Actually performs the process. By the time this method is invoked, it is known that the given URI passes the Processor.getEnabled(), the Processor.getShouldProcessRule() and the Processor.shouldProcess(CrawlURI) tests.
    
    Specified by:
    
    innerProcess in class Processor
    
    Parameters:
    
    puri - the URI to process
  - innerProcessResult
```
protected abstract ProcessResult innerProcessResult(CrawlURI uri)
```
    Overrides:
    
    innerProcessResult in class Processor
  - shouldProcess
```
protected boolean shouldProcess(CrawlURI curi)
```
    Description copied from class: Processor
    
    Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.
    
    Specified by:
    
    shouldProcess in class Processor
    
    Parameters:
    
    curi - the URI to test
    
    Returns:
    
    true if this processor should process that uri; false if not
  - copyForwardWriteTagIfDupe
```
protected void copyForwardWriteTagIfDupe(CrawlURI curi)
```
    If this fetch is identical to the last written (archived) fetch, then copy forward the writeTag. This method should generally be called when writeTag is present from a previous identical fetch, even though this particular fetch is not being written anywhere (not even a revisit record).
  - innerRejectProcess
```
protected void innerRejectProcess(CrawlURI curi)
                           throws InterruptedException
```
    Description copied from class: Processor
    
    Invoked after a URI has been rejected. The default implementation does nothing; subclasses may override to log rejects or something.
    
    Overrides:
    
    innerRejectProcess in class Processor
    
    Parameters:
    
    curi - the URI that was rejected
    
    Throws:
    
    InterruptedException - if the thread is interrupted

Class WriterPoolProcessor

Field Summary

Fields inherited from class org.archive.modules.Processor

Constructor Summary

Method Summary

Methods inherited from class org.archive.modules.Processor

Methods inherited from class java.lang.Object

Methods inherited from interface org.springframework.context.Lifecycle

Methods inherited from interface org.archive.checkpointing.Checkpointable

Field Detail

compress

prefix

template

maxFileSizeBytes

poolMaxActive

maxWaitForIdleMs

skipIdenticalDigests

ANNOTATION_UNWRITTEN

maxTotalBytesToWrite

frequentFlushes

writeBufferSize

serverCache

directory

startNewFilesOnCheckpoint

storePaths

Constructor Detail

WriterPoolProcessor

Method Detail

getCompress

setCompress

getPrefix

setPrefix

getTemplate

setTemplate

getDefaultMaxFileSize

getMaxFileSizeBytes

setMaxFileSizeBytes

getPoolMaxActive

setPoolMaxActive

getMaxWaitForIdleMs

setMaxWaitForIdleMs

getSkipIdenticalDigests

setSkipIdenticalDigests

getMaxTotalBytesToWrite

setMaxTotalBytesToWrite

getFrequentFlushes

setFrequentFlushes

getWriteBufferSize

setWriteBufferSize

getMetadataProvider

setMetadataProvider

getServerCache

setServerCache

getDirectory

setDirectory

getStartNewFilesOnCheckpoint

setStartNewFilesOnCheckpoint

getDefaultStorePaths

getStorePaths

setStorePaths

start

stop

getSerialNo

setupPool

checkBytesWritten

shouldWrite

getHostAddress

doCheckpoint

toCheckpointJson

fromCheckpointJson

getPool

setPool

getTotalBytesWritten

setTotalBytesWritten

getMetadata

calcOutputDirs

innerProcess

innerProcessResult

shouldProcess

copyForwardWriteTagIfDupe