MirrorWriterProcessor (Heritrix 3: 'modules' subproject (reusable components) 3.4.0-20190418 API)

java.lang.Object
- org.archive.modules.Processor
- - org.archive.modules.writer.MirrorWriterProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle
```
public class MirrorWriterProcessor
extends Processor
```
Processor module that writes the results of successful fetches to files on disk. Writes contents of one URI to one file on disk. The files are arranged in a directory hierarchy based on the URI paths. In that sense they mirror the file hierarchy that might exist on the servers.
There are a number of issues involved:
- URIs can have arbitrary length, but file systems have length constraints.
- URIs can contain characters that file systems prohibit.
- URI paths are case-sensitive, but some file systems are case-insensitive.
This class tries very hard to map each URI into a file system path that obeys all file system constraints and yet reasonably represents the original URI.
There would normally be a single instance of this class per Heritrix instance. This class is thread-safe; any number of threads can be in its innerProcess method at once. However, conflicts can still arise in the file system. For example, if several threads try to create the same directory at the same time, only one can win. Therefore, there should be at most one access to a server at a given time.
Author:

Howard Lee Gayle

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`A_MIRROR_PATH`
`protected boolean`	`caseSensitiveFilesystem` True if the file system is case-sensitive, like UNIX.
`protected List<String>`	`characterMap` This list is grouped in pairs.
`protected List<String>`	`contentTypeMap` This list is grouped in pairs.
`protected boolean`	`createHostDirectory` Create a subdirectory named for the host in the URI.
`protected boolean`	`createPortDirectory` Create a subdirectory named for the port in the URI.
`protected String`	`directoryFile` Implicitly append this to a URI ending with '/'.
`protected String`	`dotBegin` If a segment starts with '.', the '.' is replaced by this.
`protected String`	`dotEnd` If a directory name ends with '.' it is replaced by this.
`protected List<String>`	`hostMap` This list is grouped in pairs.
`protected int`	`maxPathLength` Maximum file system path length.
`protected int`	`maxSegLength` Maximum file system path segment length.
`protected ConfigPath`	`path` Top-level directory for mirror files.
`protected boolean`	`suffixAtEnd` If true, the suffix is placed at the end of the path, after the query (if any).
`protected String`	`tooLongDirectory` If all the directories in the URI would exceed, or come close to exceeding, the file system maximum path length, then they are all replaced by this.
`protected List<String>`	`underscoreSet` If a directory name appears (case-insensitive) in this list then an underscore is placed before it.

Fields inherited from class org.archive.modules.Processor
beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors
Constructor and Description

MirrorWriterProcessor()

Constructors
Constructor and Description
`MirrorWriterProcessor()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`getCaseSensitiveFilesystem()`
`List<String>`	`getCharacterMap()`
`List<String>`	`getContentTypeMap()`
`boolean`	`getCreateHostDirectory()`
`boolean`	`getCreatePortDirectory()`
`String`	`getDirectoryFile()`
`String`	`getDotBegin()`
`String`	`getDotEnd()`
`List<String>`	`getHostMap()`
`int`	`getMaxPathLength()`
`int`	`getMaxSegLength()`
`ConfigPath`	`getPath()`
`boolean`	`getSuffixAtEnd()`
`String`	`getTooLongDirectory()`
`List<String>`	`getUnderscoreSet()`
`protected void`	`innerProcess(CrawlURI curi)` Actually performs the process.
`void`	`setCaseSensitiveFilesystem(boolean sensitive)`
`void`	`setCharacterMap(List<String> list)`
`void`	`setContentTypeMap(List<String> list)`
`void`	`setCreateHostDirectory(boolean hostDir)`
`void`	`setCreatePortDirectory(boolean portDir)`
`void`	`setDirectoryFile(String s)`
`void`	`setDotBegin(String s)`
`void`	`setDotEnd(String s)`
`void`	`setHostMap(List<String> list)`
`void`	`setMaxPathLength(int max)`
`void`	`setMaxSegLength(int max)`
`void`	`setPath(ConfigPath s)`
`void`	`setSuffixAtEnd(boolean suffixAtEnd)`
`void`	`setTooLongDirectory(String s)`
`void`	`setUnderscoreSet(List<String> list)`
`protected boolean`	`shouldProcess(CrawlURI curi)` Determines whether the given uri should be processed by this processor.
`protected void`	`validate(Pattern pat, String s)`

Methods inherited from class org.archive.modules.Processor
doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - A_MIRROR_PATH
```
public static final String A_MIRROR_PATH
```
    See Also:
    
    Constant Field Values
  - caseSensitiveFilesystem
```
protected boolean caseSensitiveFilesystem
```
    True if the file system is case-sensitive, like UNIX. False if the file system is case-insensitive, like Macintosh HFS+ and Windows.
  - characterMap
```
protected List<String> characterMap
```
    This list is grouped in pairs. The first string in each pair must have a length of one. If it occurs in a URI path, it is replaced by the second string in the pair. For UNIX, no character mapping is normally needed. For Macintosh, the recommended value is [: %%3A]. For Windows, the recommended value is [' ' %%20 " %%22 * %%2A : %%3A < %%3C \\> %%3E ? %%3F \\\\ %%5C ^ %%5E | %%7C].
  - contentTypeMap
```
protected List<String> contentTypeMap
```
    This list is grouped in pairs. If the content type of a resource begins (case-insensitive) with the first string in a pair, the suffix is set to the second string in the pair, replacing any suffix that may have been in the URI. For example, to force all HTML files to have the same suffix, use [text/html html].
  - dotBegin
```
protected String dotBegin
```
    If a segment starts with '.', the '.' is replaced by this.
  - dotEnd
```
protected String dotEnd
```
    If a directory name ends with '.' it is replaced by this. For all file systems except Windows, '.' is recommended. For Windows, %%2E is recommended.
  - directoryFile
```
protected String directoryFile
```
    Implicitly append this to a URI ending with '/'.
  - createHostDirectory
```
protected boolean createHostDirectory
```
    Create a subdirectory named for the host in the URI.
  - hostMap
```
protected List<String> hostMap
```
    This list is grouped in pairs. If a host name matches (case-insensitive) the first string in a pair, it is replaced by the second string in the pair. This can be used for consistency when several names are used for one host, for example [12.34.56.78 www42.foo.com].
  - maxPathLength
```
protected int maxPathLength
```
    Maximum file system path length.
  - maxSegLength
```
protected int maxSegLength
```
    Maximum file system path segment length.
  - path
```
protected ConfigPath path
```
    Top-level directory for mirror files.
  - createPortDirectory
```
protected boolean createPortDirectory
```
    Create a subdirectory named for the port in the URI.
  - suffixAtEnd
```
protected boolean suffixAtEnd
```
    If true, the suffix is placed at the end of the path, after the query (if any). If false, the suffix is placed before the query.
  - tooLongDirectory
```
protected String tooLongDirectory
```
    If all the directories in the URI would exceed, or come close to exceeding, the file system maximum path length, then they are all replaced by this.
  - underscoreSet
```
protected List<String> underscoreSet
```
    If a directory name appears (case-insensitive) in this list then an underscore is placed before it. For all file systems except Windows, this is not needed. For Windows, the following is recommended: [com1 com2 com3 com4 com5 com6 com7 com8 com9 lpt1 lpt2 lpt3 lpt4 lpt5 lpt6 lpt7 lpt8 lpt9 con nul prn].
- Constructor Detail
  - MirrorWriterProcessor
```
public MirrorWriterProcessor()
```
- Method Detail
  - getCaseSensitiveFilesystem
```
public boolean getCaseSensitiveFilesystem()
```
  - setCaseSensitiveFilesystem
```
public void setCaseSensitiveFilesystem(boolean sensitive)
```
  - getCharacterMap
```
public List<String> getCharacterMap()
```
  - setCharacterMap
```
public void setCharacterMap(List<String> list)
```
  - getContentTypeMap
```
public List<String> getContentTypeMap()
```
  - setContentTypeMap
```
public void setContentTypeMap(List<String> list)
```
  - getDotBegin
```
public String getDotBegin()
```
  - setDotBegin
```
public void setDotBegin(String s)
```
  - validate
```
protected void validate(Pattern pat,
                        String s)
```
  - getDotEnd
```
public String getDotEnd()
```
  - setDotEnd
```
public void setDotEnd(String s)
```
  - getDirectoryFile
```
public String getDirectoryFile()
```
  - setDirectoryFile
```
public void setDirectoryFile(String s)
```
  - getCreateHostDirectory
```
public boolean getCreateHostDirectory()
```
  - setCreateHostDirectory
```
public void setCreateHostDirectory(boolean hostDir)
```
  - getHostMap
```
public List<String> getHostMap()
```
  - setHostMap
```
public void setHostMap(List<String> list)
```
  - getMaxPathLength
```
public int getMaxPathLength()
```
  - setMaxPathLength
```
public void setMaxPathLength(int max)
```
  - getMaxSegLength
```
public int getMaxSegLength()
```
  - setMaxSegLength
```
public void setMaxSegLength(int max)
```
  - getPath
```
public ConfigPath getPath()
```
  - setPath
```
public void setPath(ConfigPath s)
```
  - getCreatePortDirectory
```
public boolean getCreatePortDirectory()
```
  - setCreatePortDirectory
```
public void setCreatePortDirectory(boolean portDir)
```
  - getSuffixAtEnd
```
public boolean getSuffixAtEnd()
```
  - setSuffixAtEnd
```
public void setSuffixAtEnd(boolean suffixAtEnd)
```
  - getTooLongDirectory
```
public String getTooLongDirectory()
```
  - setTooLongDirectory
```
public void setTooLongDirectory(String s)
```
  - getUnderscoreSet
```
public List<String> getUnderscoreSet()
```
  - setUnderscoreSet
```
public void setUnderscoreSet(List<String> list)
```
  - shouldProcess
```
protected boolean shouldProcess(CrawlURI curi)
```
    Description copied from class: Processor
    
    Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.
    
    Specified by:
    
    shouldProcess in class Processor
    
    Parameters:
    
    curi - the URI to test
    
    Returns:
    
    true if this processor should process that uri; false if not
  - innerProcess
```
protected void innerProcess(CrawlURI curi)
```
    Description copied from class: Processor
    
    Actually performs the process. By the time this method is invoked, it is known that the given URI passes the Processor.getEnabled(), the Processor.getShouldProcessRule() and the Processor.shouldProcess(CrawlURI) tests.
    
    Specified by:
    
    innerProcess in class Processor
    
    Parameters:
    
    curi - the URI to process

Class MirrorWriterProcessor

Field Summary

Fields inherited from class org.archive.modules.Processor

Constructor Summary

Method Summary

Methods inherited from class org.archive.modules.Processor

Methods inherited from class java.lang.Object

Field Detail

A_MIRROR_PATH

caseSensitiveFilesystem

characterMap

contentTypeMap

dotBegin

dotEnd

directoryFile

createHostDirectory

hostMap

maxPathLength

maxSegLength

path

createPortDirectory

suffixAtEnd

tooLongDirectory

underscoreSet

Constructor Detail

MirrorWriterProcessor

Method Detail

getCaseSensitiveFilesystem

setCaseSensitiveFilesystem

getCharacterMap

setCharacterMap

getContentTypeMap

setContentTypeMap

getDotBegin

setDotBegin

validate

getDotEnd

setDotEnd

getDirectoryFile

setDirectoryFile

getCreateHostDirectory

setCreateHostDirectory

getHostMap

setHostMap

getMaxPathLength

setMaxPathLength

getMaxSegLength

setMaxSegLength

getPath

setPath

getCreatePortDirectory

setCreatePortDirectory

getSuffixAtEnd

setSuffixAtEnd

getTooLongDirectory

setTooLongDirectory

getUnderscoreSet

setUnderscoreSet

shouldProcess

innerProcess