KeyedProperties kp
Map<K,V> availableRobotsPolicies
String operator
String description
String audience
String organization
String jobName
private void readObject(ObjectInputStream stream) throws IOException, ClassNotFoundException
IOException
ClassNotFoundException
private void writeObject(ObjectOutputStream stream) throws IOException
IOException
UURI uuri
boolean isSeed
String pathFromSeed
P precondition R redirection E embedded (as frame, src, link, codebase, etc.) X speculative embed (as from javascript, some alternate-format extractors L linkFor example LLLE (an embedded image on a page 3 links from seed).
UURI via
LinkContext viaContext
int schedulingDirective
String classKey
int precedence
int fetchStatus
int deferrals
int fetchAttempts
String userAgent
long contentSize
long contentLength
Map<K,V> data
The attribute list is a flexible map of key/value pairs for storing
status of this URI for use by other processors. By convention the
attribute list is keyed by constants found in the
CoreAttributeConstants
interface. Use this list to carry
data or state produced by custom processors rather change the
classes CrawlURI
or this class, CrawlURI.
boolean forceRevisit
String contentType
boolean prerequisite
CrawlURI.FetchType fetchType
long ordinal
byte[] contentDigest
String contentDigestScheme
int holderCost
String canonicalString
long politenessDelay
long rescheduleTime
org.json.JSONObject extraInfo
KeyedProperties kp
String domain
KeyedProperties kp
String loginUri
Map<K,V> formItems
HtmlFormCredential.Method httpMethod
KeyedProperties kp
String comment
boolean logExtraInfo
SimpleFileLoggerProvider loggerModule
ServerCache serverCache
String beanName
boolean isRunning
ExternalGeoLookupInterface lookup
List<E> countryCodes
ServerCache serverCache
String engineName
ReadSource scriptSource
boolean isolateThreads
org.springframework.context.ApplicationContext appCtx
SurtPrefixSet surtPrefixes
ReadSource surtsSource
boolean seedsAsSurtPrefixes
ConfigFile surtsDumpFile
SeedModule seeds
SurtPrefixSet surtPrefixes
String beanName
Checkpoint recoveryCheckpoint
String path
String desc
ObjectIdentityCache<V extends IdentityCacheable> servers
ObjectIdentityCache<V extends IdentityCacheable> hosts
long lastSuccessTime
BdbModule bdb
boolean isRunning
boolean isCheckpointRecovery
String hostname
String countryCode
InetAddress ip
long ipFetched
FetchStats substats
long ipTTL
TTL a 32 bit unsigned integer that specifies the time interval (in seconds) that the resource record may be cached before it should be discarded. Zero values are interpreted to mean that the RR can only be used for the transaction in progress, and should not be cached.
long earliestNextURIEmitTime
String server
int port
Robotstxt robotstxt
long robotsFetched
boolean validRobots
FetchStats substats
int consecutiveConnectionErrors
ConcurrentSkipListSet<E> disallows
ConcurrentSkipListSet<E> allows
float crawlDelay
LinkedList<E> namedUserAgents
Map<K,V> agentsToDirectives
RobotsDirectives wildcardDirectives
boolean hasErrors
boolean sourceTagSeeds
Set<E> seedListeners
ReadSource textSource
int blockAwaitingSeedLines
Copyright © 2003–2021 Internet Archive. All rights reserved.