A B C D E F G H I J K L M N O P R S T U V W

A

AbstractQueueScheduler - Class in ai.preferred.venom.job

AbstractQueueScheduler() - Constructor for class ai.preferred.venom.job.AbstractQueueScheduler

add(Request, Handleable, Priority) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

add(Request, Handleable) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

add(Request, Priority, Priority) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

add(Request, Priority) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

add(Request) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

add(Request, Handleable, Priority, Priority) - Method in class ai.preferred.venom.job.FIFOScheduler

add(Request, Handleable, Priority, Priority) - Method in class ai.preferred.venom.job.LazyScheduler

add(Request, Handleable, Priority, Priority) - Method in class ai.preferred.venom.job.PriorityQueueScheduler

add(Request, Handleable, Priority, Priority) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(Request, Handleable, Priority) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(Request, Handleable) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(Request, Priority, Priority) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(Request, Priority) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(Request) - Method in interface ai.preferred.venom.job.Scheduler

Adds a request to the queue

add(HttpHost) - Method in interface ai.preferred.venom.ProxyProvider

Add a proxy to the list

addAll(Collection<HttpHost>) - Method in interface ai.preferred.venom.ProxyProvider

Add a list of proxies to the list

addHeader(String, String) - Method in class ai.preferred.venom.request.VRequest.Builder

Adds header to be used

addHeaders(Map<String, String>) - Method in class ai.preferred.venom.request.VRequest.Builder

Add headers to be used

ai.preferred.venom - package ai.preferred.venom

ai.preferred.venom.fetcher - package ai.preferred.venom.fetcher

ai.preferred.venom.job - package ai.preferred.venom.job

ai.preferred.venom.request - package ai.preferred.venom.request

ai.preferred.venom.response - package ai.preferred.venom.response

ai.preferred.venom.storage - package ai.preferred.venom.storage

ai.preferred.venom.uagent - package ai.preferred.venom.uagent

ai.preferred.venom.utils - package ai.preferred.venom.utils

ai.preferred.venom.validator - package ai.preferred.venom.validator

ALWAYS_VALID - Static variable in interface ai.preferred.venom.validator.Validator

AsyncFetcher - Class in ai.preferred.venom.fetcher

This class holds the implementation to provide how items are fetched, to fetch the item, to validate the item and to store it if specified.

AsyncFetcher.Builder - Class in ai.preferred.venom.fetcher

AsyncResponseConsumer - Class in ai.preferred.venom.fetcher

On top of the abstract class, this class handles the parsing of a response from the web service.

AsyncResponseConsumer(Validator, Set<Integer>, boolean, HttpFetcherRequest) - Constructor for class ai.preferred.venom.fetcher.AsyncResponseConsumer

awaitTermination(long, TimeUnit) - Method in class ai.preferred.venom.utils.InlineExecutorService

B

BaseResponse - Class in ai.preferred.venom.response

BaseResponse(int, String, byte[], ContentType, Header[], HttpHost) - Constructor for class ai.preferred.venom.response.BaseResponse

BaseResponse(int, String, byte[], ContentType, Header[], HttpHost, Validator, String) - Constructor for class ai.preferred.venom.response.BaseResponse

BasicJob - Class in ai.preferred.venom.job

BasicJob(Request, Handleable, Priority, Priority, Queue<Job>) - Constructor for class ai.preferred.venom.job.BasicJob

build() - Method in class ai.preferred.venom.Crawler.Builder

Builds the crawler with the options specified.

build() - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Builds the fetcher with the options specified.

build(Request.Method, String) - Static method in class ai.preferred.venom.request.VRequest

build() - Method in class ai.preferred.venom.request.VRequest.Builder

Builds the request with the options specified.

build() - Method in class ai.preferred.venom.Session.Builder

buildDefault() - Static method in class ai.preferred.venom.Crawler

Builds a new default instance of Crawler.

buildDefault() - Static method in class ai.preferred.venom.fetcher.AsyncFetcher

builder() - Static method in class ai.preferred.venom.Crawler

Creates a new instance of Builder.

builder() - Static method in class ai.preferred.venom.fetcher.AsyncFetcher

Builder(Request.Method, String) - Constructor for class ai.preferred.venom.request.VRequest.Builder

builder() - Static method in class ai.preferred.venom.Session

Builder() - Constructor for class ai.preferred.venom.Session.Builder

buildResult(HttpContext) - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer

C

Callback - Interface in ai.preferred.venom.fetcher

cancel(boolean) - Method in class ai.preferred.venom.job.BasicJob

cancel(boolean) - Method in interface ai.preferred.venom.job.Job

Attempts to cancel execution of this task.

cancelled() - Method in class ai.preferred.venom.Crawler.AsyncCrawlerCallbackProcessor

cancelled(Request) - Method in interface ai.preferred.venom.fetcher.Callback

Method to be call upon cancellation of request.

close() - Method in class ai.preferred.venom.Crawler

close() - Method in class ai.preferred.venom.fetcher.AsyncFetcher

close() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

close() - Method in class ai.preferred.venom.ThreadedWorkerManager

compareTo(Job) - Method in class ai.preferred.venom.job.BasicJob

completed(Response) - Method in class ai.preferred.venom.Crawler.AsyncCrawlerCallbackProcessor

completed(Request, Response) - Method in interface ai.preferred.venom.fetcher.Callback

Method to be call upon completion of request.

compressed(boolean) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Set whether to request for compress pages and to decompress pages after it is fetched.

connectionRequestTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

The timeout in milliseconds used when requesting a connection from the connection manager.

connectTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Determines the timeout in milliseconds until a connection is established.

Crawler - Class in ai.preferred.venom

This class handles the coordination between classes during the pre and post fetching of a page such as executing threads, calling to fetcher and manipulating the priority of a scheduled request.

Crawler.AsyncCrawlerCallbackProcessor - Class in ai.preferred.venom

Crawler.Builder - Class in ai.preferred.venom

A builder for Crawler class

CrawlerRequest - Class in ai.preferred.venom.request

CrawlerRequest(Request) - Constructor for class ai.preferred.venom.request.CrawlerRequest

D

decompress(HttpResponse) - Method in class ai.preferred.venom.utils.ResponseDecompressor

DEFAULT - Static variable in enum ai.preferred.venom.job.Priority

DEFAULT_CHARSET - Static variable in class ai.preferred.venom.response.VResponse

DefaultUserAgent - Class in ai.preferred.venom.uagent

DefaultUserAgent() - Constructor for class ai.preferred.venom.uagent.DefaultUserAgent

delete(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type delete.

done() - Method in class ai.preferred.venom.job.BasicJob

done() - Method in interface ai.preferred.venom.job.Job

Set done to true and interrupts all the threads

downgrade(Priority) - Method in enum ai.preferred.venom.job.Priority

downgrade() - Method in enum ai.preferred.venom.job.Priority

Returns the priority one level below the current priority, if priority is higher than the default floor or the lowest available priority.

drainTo(Collection<? super Job>) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

drainTo(Collection<? super Job>, int) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

E

EMPTY_PROXY_PROVIDER - Static variable in interface ai.preferred.venom.ProxyProvider

EMPTY_SESSION - Static variable in class ai.preferred.venom.Session

EmptyContentValidator - Class in ai.preferred.venom.validator

EmptyContentValidator() - Constructor for class ai.preferred.venom.validator.EmptyContentValidator

execute(Runnable) - Method in class ai.preferred.venom.utils.InlineExecutorService

F

failed(Exception) - Method in class ai.preferred.venom.Crawler.AsyncCrawlerCallbackProcessor

failed(Request, Exception) - Method in interface ai.preferred.venom.fetcher.Callback

Method to be call upon failure of request.

fetch(Request) - Method in class ai.preferred.venom.fetcher.AsyncFetcher

fetch(Request, FutureCallback<Response>) - Method in class ai.preferred.venom.fetcher.AsyncFetcher

fetch(Request) - Method in interface ai.preferred.venom.fetcher.Fetcher

Fetch the desired HTTP page given in Request.

fetch(Request, FutureCallback<Response>) - Method in interface ai.preferred.venom.fetcher.Fetcher

Fetch the desired HTTP page given in Request.

fetcher(Fetcher) - Method in class ai.preferred.venom.Crawler.Builder

Sets the Fetcher to be used, if not set, default will be chosen.

Fetcher - Interface in ai.preferred.venom.fetcher

This interface represents only the most basic of fetching a request.

FIFOScheduler - Class in ai.preferred.venom.job

FIFOScheduler() - Constructor for class ai.preferred.venom.job.FIFOScheduler

fileManager(FileManager) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets the FileManager to be used.

FileManager - Interface in ai.preferred.venom.storage

fillInStackTrace() - Method in exception ai.preferred.venom.fetcher.StopCodeException

fillInStackTrace() - Method in exception ai.preferred.venom.fetcher.ValidationException

FLOOR - Static variable in enum ai.preferred.venom.job.Priority

G

get(Request) - Method in interface ai.preferred.venom.ProxyProvider

Returns the get proxy from the list

get(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type get.

get(Session.Key<T>) - Method in class ai.preferred.venom.Session

Returns the session variable from the store

get(int) - Method in interface ai.preferred.venom.storage.FileManager

Returns record by the internal record id.

get(Request) - Method in interface ai.preferred.venom.storage.FileManager

Returns latest record matching request.

get() - Method in class ai.preferred.venom.uagent.DefaultUserAgent

get() - Method in interface ai.preferred.venom.uagent.UserAgent

Gets the user agent for a request

getBaseUrl() - Method in class ai.preferred.venom.response.BaseResponse

getBaseUrl() - Method in interface ai.preferred.venom.response.Response

Returns the base form of the url used in this request

getBaseUrl() - Method in class ai.preferred.venom.response.VResponse

getBody() - Method in class ai.preferred.venom.request.CrawlerRequest

getBody() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getBody() - Method in interface ai.preferred.venom.request.Request

Returns the request body of the request or null if none specified

getBody() - Method in class ai.preferred.venom.request.VRequest

getCallback() - Method in interface ai.preferred.venom.storage.FileManager

Get callback upon completion of request.

getContent() - Method in class ai.preferred.venom.response.BaseResponse

getContent() - Method in interface ai.preferred.venom.response.Response

Returns raw content of the response

getContent() - Method in class ai.preferred.venom.response.VResponse

getContentType() - Method in class ai.preferred.venom.response.BaseResponse

getContentType() - Method in interface ai.preferred.venom.response.Response

Returns the content type of the content fetched

getContentType() - Method in class ai.preferred.venom.response.VResponse

getDateCreated() - Method in interface ai.preferred.venom.storage.Record

getHandler(Request) - Method in interface ai.preferred.venom.HandlerRouter

Returns the handler to be used for a specified request

getHandler() - Method in class ai.preferred.venom.job.BasicJob

getHandler() - Method in interface ai.preferred.venom.job.Job

Get the handler to handle the response of the job.

getHandler(Request) - Method in class ai.preferred.venom.UrlRouter

getHeaders() - Method in class ai.preferred.venom.request.CrawlerRequest

getHeaders() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getHeaders() - Method in interface ai.preferred.venom.request.Request

Returns the headers set for the request

getHeaders() - Method in class ai.preferred.venom.request.VRequest

getHeaders() - Method in class ai.preferred.venom.response.BaseResponse

getHeaders() - Method in interface ai.preferred.venom.response.Response

Returns the headers that were used to trigger this response

getHeaders() - Method in class ai.preferred.venom.response.VResponse

getHtml() - Method in class ai.preferred.venom.response.VResponse

Returns the html in string format

getHtml(Charset) - Method in class ai.preferred.venom.response.VResponse

Returns the html in string format

getId() - Method in interface ai.preferred.venom.storage.Record

getInner() - Method in class ai.preferred.venom.request.CrawlerRequest

getInner() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getInner() - Method in interface ai.preferred.venom.request.Unwrappable

Returns the unwrapped version of this request

getInner() - Method in interface ai.preferred.venom.response.Unwrappable

Returns the unwrapped version of this request

getInner() - Method in class ai.preferred.venom.response.VResponse

getJsoup() - Method in class ai.preferred.venom.response.VResponse

Returns a Jsoup document of this response

getJsoup(Charset) - Method in class ai.preferred.venom.response.VResponse

Returns a Jsoup document of this response

getMD5() - Method in interface ai.preferred.venom.storage.Record

getMethod() - Method in class ai.preferred.venom.request.CrawlerRequest

getMethod() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getMethod() - Method in interface ai.preferred.venom.request.Request

Returns the method type of the request

getMethod() - Method in class ai.preferred.venom.request.VRequest

getPriority() - Method in class ai.preferred.venom.job.BasicJob

getPriority() - Method in interface ai.preferred.venom.job.Job

Get the current priority set for this job.

getPriority() - Method in enum ai.preferred.venom.job.Priority

getProxy() - Method in class ai.preferred.venom.request.CrawlerRequest

getProxy() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getProxy() - Method in interface ai.preferred.venom.request.Request

Returns the proxy set to be used for the request or default to fetcher if none specified

getProxy() - Method in class ai.preferred.venom.request.VRequest

getProxy() - Method in class ai.preferred.venom.response.BaseResponse

getProxy() - Method in interface ai.preferred.venom.response.Response

Returns the proxy that was used to trigger this response

getProxy() - Method in class ai.preferred.venom.response.VResponse

getProxyList() - Method in interface ai.preferred.venom.ProxyProvider

Returns a list of all proxies

getRequest() - Method in class ai.preferred.venom.job.BasicJob

getRequest() - Method in interface ai.preferred.venom.job.Job

Get the request of this job.

getRequestBody() - Method in interface ai.preferred.venom.storage.Record

getRequestHeaders() - Method in interface ai.preferred.venom.storage.Record

getRequestMethod() - Method in interface ai.preferred.venom.storage.Record

getResolvedHtml() - Method in class ai.preferred.venom.response.VResponse

Returns the html in string format with all relative urls resolved to absolute urls

getResolvedHtml(Charset) - Method in class ai.preferred.venom.response.VResponse

Returns the html in string format with all relative urls resolved to absolute urls

getResponse() - Method in exception ai.preferred.venom.fetcher.ValidationException

getResponseContent() - Method in interface ai.preferred.venom.storage.Record

getResponseHeaders() - Method in interface ai.preferred.venom.storage.Record

getScheduler() - Method in class ai.preferred.venom.Crawler

Get the instance of scheduler used.

getSleepScheduler() - Method in class ai.preferred.venom.request.CrawlerRequest

getSleepScheduler() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getSleepScheduler() - Method in interface ai.preferred.venom.request.Request

Returns information about the amount of sleep before this request is made

getSleepScheduler() - Method in class ai.preferred.venom.request.VRequest

getSleepTime() - Method in class ai.preferred.venom.SleepScheduler

Get the amount of time to wait specified in this class

getSourceId() - Method in class ai.preferred.venom.response.BaseResponse

getSourceId() - Method in interface ai.preferred.venom.response.Retrievable

Returns the id of the row where an archive of this response has been insert into a persistent storage

getStatus() - Method in exception ai.preferred.venom.fetcher.ValidationException

getStatusCode() - Method in class ai.preferred.venom.response.BaseResponse

getStatusCode() - Method in interface ai.preferred.venom.response.Response

Returns status code of the response

getStatusCode() - Method in class ai.preferred.venom.response.VResponse

getStatusCode() - Method in interface ai.preferred.venom.storage.Record

getStreamResponseContent() - Method in interface ai.preferred.venom.storage.Record

getTryCount() - Method in class ai.preferred.venom.job.BasicJob

getTryCount() - Method in interface ai.preferred.venom.job.Job

Get attempt number of this job.

getUrl() - Method in class ai.preferred.venom.request.CrawlerRequest

getUrl() - Method in class ai.preferred.venom.request.HttpFetcherRequest

getUrl() - Method in interface ai.preferred.venom.request.Request

Returns the url of the request

getUrl() - Method in class ai.preferred.venom.request.VRequest

getURL() - Method in interface ai.preferred.venom.storage.Record

getValidator() - Method in class ai.preferred.venom.response.BaseResponse

getValidator() - Method in interface ai.preferred.venom.response.Response

Returns the instance of validator used to validate this response

getValidator() - Method in class ai.preferred.venom.response.VResponse

getValidator(Request) - Method in class ai.preferred.venom.UrlRouter

getValidator(Request) - Method in interface ai.preferred.venom.ValidatorRouter

Returns the handler to be used for a specified request

getWorker() - Method in class ai.preferred.venom.ThreadedWorkerManager

getWorker() - Method in interface ai.preferred.venom.WorkerManager

Get the result collector in use

H

handle(Request, Response, Scheduler, Session, Worker) - Method in interface ai.preferred.venom.Handleable

This function is called when the request is fetched successfully.

handle(Request, Response, Scheduler, Session, Worker) - Method in interface ai.preferred.venom.Handler

handle(Request, VResponse, Scheduler, Session, Worker) - Method in interface ai.preferred.venom.Handler

This function is called when the request is fetched successfully.

Handleable - Interface in ai.preferred.venom

This interface represents the method call when the response is valid the parsing logic will be up to individual implementations.

Handler - Interface in ai.preferred.venom

This interface represents the method call when the response is makeValidResponse the parsing logic will be up to individual implementations.

HandlerRouter - Interface in ai.preferred.venom

This interface allows the user to map request to handleable

head(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type head.

headers(Map<String, String>) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets the headers to be used when fetching items.

HttpFetcherRequest - Class in ai.preferred.venom.request

HttpFetcherRequest(Request) - Constructor for class ai.preferred.venom.request.HttpFetcherRequest

I

InlineExecutorService - Class in ai.preferred.venom.utils

InlineExecutorService() - Constructor for class ai.preferred.venom.utils.InlineExecutorService

INSTANCE - Static variable in class ai.preferred.venom.validator.EmptyContentValidator

INSTANCE - Static variable in class ai.preferred.venom.validator.StatusOkValidator

interruptAndClose() - Method in class ai.preferred.venom.Crawler

interruptAndClose() - Method in interface ai.preferred.venom.Interruptible

Interrupt a thread and then close it.

interruptAndClose() - Method in class ai.preferred.venom.ThreadedWorkerManager

Interruptible - Interface in ai.preferred.venom

isCancelled() - Method in class ai.preferred.venom.job.BasicJob

isCancelled() - Method in interface ai.preferred.venom.job.Job

Returns true if this task was cancelled before it completed normally.

isDone() - Method in class ai.preferred.venom.job.BasicJob

isDone() - Method in interface ai.preferred.venom.job.Job

Returns true if this task completed.

isEmpty() - Method in class ai.preferred.venom.job.LazyScheduler

isShutdown() - Method in class ai.preferred.venom.utils.InlineExecutorService

isTerminated() - Method in class ai.preferred.venom.utils.InlineExecutorService

isValid(Request, Response) - Method in class ai.preferred.venom.validator.EmptyContentValidator

isValid(Request, Response) - Method in class ai.preferred.venom.validator.MimeTypeValidator

isValid(Request, Response) - Method in class ai.preferred.venom.validator.PipelineValidator

isValid(Request, Response) - Method in class ai.preferred.venom.validator.StatusOkValidator

isValid(Request, Response) - Method in interface ai.preferred.venom.validator.Validator

Method will be called when a response need validation

iterator() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

J

Job - Interface in ai.preferred.venom.job

This interface represents only the most basic of a job, to be placed in a scheduler or other forms.

K

Key() - Constructor for class ai.preferred.venom.Session.Key

L

LazyScheduler - Class in ai.preferred.venom.job

LazyScheduler(Iterator<Request>, Handler) - Constructor for class ai.preferred.venom.job.LazyScheduler

LazyScheduler(Iterator<Request>) - Constructor for class ai.preferred.venom.job.LazyScheduler

M

maxConnections(int) - Method in class ai.preferred.venom.Crawler.Builder

The number of concurrent connections allowed out of the client.

maxTries(int) - Method in class ai.preferred.venom.Crawler.Builder

Sets number of times to retry for a request.

MimeTypeValidator - Class in ai.preferred.venom.validator

MimeTypeValidator(String) - Constructor for class ai.preferred.venom.validator.MimeTypeValidator

MimeTypeValidator(Pattern) - Constructor for class ai.preferred.venom.validator.MimeTypeValidator

N

name(String) - Method in class ai.preferred.venom.Crawler.Builder

Sets the name for crawler thread

numIoThreads(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Number of httpclient dispatcher threads.

O

offer(Job) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

offer(Job, long, TimeUnit) - Method in class ai.preferred.venom.job.FIFOScheduler

offer(Job, long, TimeUnit) - Method in class ai.preferred.venom.job.LazyScheduler

offer(Job, long, TimeUnit) - Method in class ai.preferred.venom.job.PriorityQueueScheduler

onContentReceived(ContentDecoder, IOControl) - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer

onEntityEnclosed(HttpEntity, ContentType) - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer

onResponseReceived(HttpResponse) - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer

options(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type options.

P

parallism(int) - Method in class ai.preferred.venom.Crawler.Builder

Sets the parallelism level.

peek() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

PipelineValidator - Class in ai.preferred.venom.validator

PipelineValidator(Validator...) - Constructor for class ai.preferred.venom.validator.PipelineValidator

PipelineValidator(List<Validator>) - Constructor for class ai.preferred.venom.validator.PipelineValidator

poll(long, TimeUnit) - Method in class ai.preferred.venom.job.AbstractQueueScheduler

poll() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

poll() - Method in class ai.preferred.venom.job.LazyScheduler

poll(long, TimeUnit) - Method in class ai.preferred.venom.job.LazyScheduler

post(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type post.

prependHeaders(Map<String, String>) - Method in class ai.preferred.venom.request.HttpFetcherRequest

Priority - Enum in ai.preferred.venom.job

Job priorities, list in descending order of priority:

PriorityQueueScheduler - Class in ai.preferred.venom.job

PriorityQueueScheduler() - Constructor for class ai.preferred.venom.job.PriorityQueueScheduler

propRetainProxy(double) - Method in class ai.preferred.venom.Crawler.Builder

Sets the proportion of max tries where a specified proxy, if specified will be used.

proxyProvider(ProxyProvider) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets the ProxyProvider to be used.

ProxyProvider - Interface in ai.preferred.venom

put(Job) - Method in class ai.preferred.venom.job.FIFOScheduler

put(Job) - Method in class ai.preferred.venom.job.LazyScheduler

put(Job) - Method in class ai.preferred.venom.job.PriorityQueueScheduler

put(String) - Static method in class ai.preferred.venom.request.VRequest.Builder

Creates a new instance of builder with method type put.

put(Session.Key<T>, T) - Method in class ai.preferred.venom.Session.Builder

Adds a session variable into store

put(Request, Response) - Method in interface ai.preferred.venom.storage.FileManager

Puts record into database.

R

Record - Interface in ai.preferred.venom.storage

register(Callback) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Register any callbacks that will be called when a page has been fetched.

register(Pattern, Handler) - Method in class ai.preferred.venom.UrlRouter

Adds a url pattern, and the handler to be used.

register(Pattern, Validator) - Method in class ai.preferred.venom.UrlRouter

Adds a url pattern, and the handler to be used.

register(Pattern, Handler, Validator) - Method in class ai.preferred.venom.UrlRouter

Adds a url pattern, and the handler to be used.

releaseResources() - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer

remainingCapacity() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

remove(HttpHost) - Method in interface ai.preferred.venom.ProxyProvider

Remove a proxy from the list

removeHeader(String) - Method in class ai.preferred.venom.request.VRequest.Builder

removeHeaders() - Method in class ai.preferred.venom.request.VRequest.Builder

removeProxy() - Method in class ai.preferred.venom.request.CrawlerRequest

Request - Interface in ai.preferred.venom.request

Request.Method - Enum in ai.preferred.venom.request

The method of the request to be made

reQueue() - Method in class ai.preferred.venom.job.BasicJob

reQueue() - Method in interface ai.preferred.venom.job.Job

Remove any existing in queue, downgrades the priority and adds the job back into queue.

resolveUrls(String, String) - Static method in class ai.preferred.venom.utils.UrlUtils

Response - Interface in ai.preferred.venom.response

ResponseDecompressor - Class in ai.preferred.venom.utils

Modified from: org.apache.http.client.protocol.ResponseContentEncoding

ResponseDecompressor() - Constructor for class ai.preferred.venom.utils.ResponseDecompressor

Retrievable - Interface in ai.preferred.venom.response

This interface represents that the response can be/ has been stored

router(HandlerRouter) - Method in class ai.preferred.venom.Crawler.Builder

Sets HandlerRouter to be used.

router(ValidatorRouter) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets ValidatorRouter to be used.

S

scheduler(AbstractQueueScheduler) - Method in class ai.preferred.venom.Crawler.Builder

Sets the Scheduler to be used, if not set, default will be chosen.

Scheduler - Interface in ai.preferred.venom.job

This interface represents only the most basic of a scheduler.

session(Session) - Method in class ai.preferred.venom.Crawler.Builder

Sets the Session to be used, if not set, defaults to none.

Session - Class in ai.preferred.venom

Session.Builder - Class in ai.preferred.venom

Builder for Session

Session.Key<T> - Class in ai.preferred.venom

A class representing the key for a session

setBody(String) - Method in class ai.preferred.venom.request.VRequest.Builder

Sets the request body to be used.

setProxy(HttpHost) - Method in class ai.preferred.venom.request.HttpFetcherRequest

setProxy(HttpHost) - Method in class ai.preferred.venom.request.VRequest.Builder

Sets the proxy to be used, this will override the proxy selected in Fetcher for this request.

setSleepScheduler(SleepScheduler) - Method in class ai.preferred.venom.request.VRequest.Builder

Sets the sleep scheduler to be used, this will override the sleep scheduler defined in Crawler for this request.

setSourceId(String) - Method in class ai.preferred.venom.response.BaseResponse

setUrl(String) - Method in class ai.preferred.venom.request.VRequest.Builder

Sets the url to be fetched.

setValidator(Validator) - Method in class ai.preferred.venom.response.BaseResponse

shutdown() - Method in class ai.preferred.venom.utils.InlineExecutorService

shutdownNow() - Method in class ai.preferred.venom.utils.InlineExecutorService

size() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

sleepScheduler(SleepScheduler) - Method in class ai.preferred.venom.Crawler.Builder

Sets the SleepScheduler to be used, if not set, default will be chosen.

SleepScheduler - Class in ai.preferred.venom

SleepScheduler(long) - Constructor for class ai.preferred.venom.SleepScheduler

SleepScheduler(long, long) - Constructor for class ai.preferred.venom.SleepScheduler

socketTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).

soTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Determines the default socket timeout value for non-blocking I/O operations.

start() - Method in class ai.preferred.venom.Crawler

Starts the crawler by starting a new thread to poll for jobs.

start() - Method in class ai.preferred.venom.fetcher.AsyncFetcher

start() - Method in interface ai.preferred.venom.fetcher.Fetcher

Fetcher starter.

startAndClose() - Method in class ai.preferred.venom.Crawler

Starts the crawler by starting a new thread to poll for jobs and close it after the queue has reached 0.

StatusOkValidator - Class in ai.preferred.venom.validator

StatusOkValidator() - Constructor for class ai.preferred.venom.validator.StatusOkValidator

StopCodeException - Exception in ai.preferred.venom.fetcher

StopCodeException(int, String) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException

StopCodeException(int, String, Throwable) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException

StopCodeException(int, Throwable) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException

StopCodeException(int, String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException

stopCodes(int...) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

StorageException - Exception in ai.preferred.venom.storage

StorageException(String) - Constructor for exception ai.preferred.venom.storage.StorageException

StorageException(String, Throwable) - Constructor for exception ai.preferred.venom.storage.StorageException

StorageException(Throwable) - Constructor for exception ai.preferred.venom.storage.StorageException

StorageException(String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.storage.StorageException

submit(Callable<T>) - Method in interface ai.preferred.venom.Worker

Submits a value-returning task for execution and returns a Future representing the pending results of the task.

submit(Runnable, T) - Method in interface ai.preferred.venom.Worker

Submits a Runnable task for execution and returns a Future representing that task.

submit(Runnable) - Method in interface ai.preferred.venom.Worker

Submits a Runnable task for execution and returns a Future representing that task.

T

take() - Method in class ai.preferred.venom.job.AbstractQueueScheduler

ThreadedWorkerManager - Class in ai.preferred.venom

ThreadedWorkerManager(int) - Constructor for class ai.preferred.venom.ThreadedWorkerManager

ThreadedWorkerManager(ExecutorService) - Constructor for class ai.preferred.venom.ThreadedWorkerManager

threadFactory(ThreadFactory) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Set the thread factory that creates the httpclient dispatcher threads.

U

Unwrappable - Interface in ai.preferred.venom.request

This interface represents that the request can be unwrapped

Unwrappable - Interface in ai.preferred.venom.response

This interface represents that the response can be unwrapped

unwrapRequest(Request) - Static method in interface ai.preferred.venom.request.Unwrappable

Unwrap all wrapped request to an instance of base request.

UrlRouter - Class in ai.preferred.venom

This class provides an implementation to select a handler based on the url from which they were fetched.

UrlRouter(Handler) - Constructor for class ai.preferred.venom.UrlRouter

UrlRouter() - Constructor for class ai.preferred.venom.UrlRouter

UrlUtils - Class in ai.preferred.venom.utils

Borrowed from WebMagic Project: https://github.com/code4craft/webmagic

userAgent(UserAgent) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets the UserAgent to be used, if not set, default will be chosen.

UserAgent - Interface in ai.preferred.venom.uagent

V

ValidationException - Exception in ai.preferred.venom.fetcher

ValidationException(Validator.Status, Response, String) - Constructor for exception ai.preferred.venom.fetcher.ValidationException

ValidationException(Validator.Status, Response, String, Throwable) - Constructor for exception ai.preferred.venom.fetcher.ValidationException

ValidationException(Validator.Status, Response, Throwable) - Constructor for exception ai.preferred.venom.fetcher.ValidationException

ValidationException(Validator.Status, Response, String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.fetcher.ValidationException

validator(Validator) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder

Sets the Validator to be used.

Validator - Interface in ai.preferred.venom.validator

Validator.Status - Enum in ai.preferred.venom.validator

The allowed return status of validation

ValidatorRouter - Interface in ai.preferred.venom

This interface allows the user to map request to handleable

valueOf(String) - Static method in enum ai.preferred.venom.job.Priority

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum ai.preferred.venom.request.Request.Method

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum ai.preferred.venom.validator.Validator.Status

Returns the enum constant of this type with the specified name.

values() - Static method in enum ai.preferred.venom.job.Priority

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum ai.preferred.venom.request.Request.Method

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum ai.preferred.venom.validator.Validator.Status

Returns an array containing the constants of this enum type, in the order they are declared.

VRequest - Class in ai.preferred.venom.request

An implementation of HTTP request.

VRequest(String) - Constructor for class ai.preferred.venom.request.VRequest

VRequest(String, Map<String, String>) - Constructor for class ai.preferred.venom.request.VRequest

VRequest(VRequest.Builder<?>) - Constructor for class ai.preferred.venom.request.VRequest

VRequest.Builder<T extends VRequest.Builder<T>> - Class in ai.preferred.venom.request

A builder for VRequest class

VResponse - Class in ai.preferred.venom.response

VResponse(Response) - Constructor for class ai.preferred.venom.response.VResponse

W

Worker - Interface in ai.preferred.venom

workerManager(WorkerManager) - Method in class ai.preferred.venom.Crawler.Builder

Sets the WorkerManager to be used, if not set, default will be chosen.

WorkerManager - Interface in ai.preferred.venom

A B C D E F G H I J K L M N O P R S T U V W