- get(Request) - Method in interface ai.preferred.venom.ProxyProvider
-
Returns the get proxy from the list.
- get(String) - Static method in class ai.preferred.venom.request.VRequest.Builder
-
Creates a new instance of builder with method type get.
- get(Session.Key<T>) - Method in class ai.preferred.venom.Session
-
Returns the session variable from the store.
- get(int) - Method in interface ai.preferred.venom.storage.FileManager
-
Returns record by the internal record id.
- get(Request) - Method in interface ai.preferred.venom.storage.FileManager
-
Returns latest record matching request.
- get() - Method in class ai.preferred.venom.uagent.DefaultUserAgent
-
- get() - Method in interface ai.preferred.venom.uagent.UserAgent
-
Gets the user agent for a request.
- getBaseUrl() - Method in class ai.preferred.venom.response.BaseResponse
-
- getBaseUrl() - Method in interface ai.preferred.venom.response.Response
-
Returns the base form of the url used in this request.
- getBaseUrl() - Method in class ai.preferred.venom.response.VResponse
-
- getBody() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getBody() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getBody() - Method in interface ai.preferred.venom.request.Request
-
Returns the request body of the request or null if none specified.
- getBody() - Method in class ai.preferred.venom.request.VRequest
-
- getCallback() - Method in interface ai.preferred.venom.storage.FileManager
-
Get callback upon completion of request.
- getContent() - Method in class ai.preferred.venom.response.BaseResponse
-
- getContent() - Method in interface ai.preferred.venom.response.Response
-
Returns raw content of the response.
- getContent() - Method in class ai.preferred.venom.response.VResponse
-
- getContentType(HttpEntity) - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer
-
- getContentType() - Method in class ai.preferred.venom.response.BaseResponse
-
- getContentType() - Method in interface ai.preferred.venom.response.Response
-
Returns the content type of the content fetched.
- getContentType() - Method in class ai.preferred.venom.response.VResponse
-
- getDateCreated() - Method in interface ai.preferred.venom.storage.Record
-
- getHandler(Request) - Method in interface ai.preferred.venom.HandlerRouter
-
Returns the handler to be used for a specified request.
- getHandler() - Method in class ai.preferred.venom.job.BasicJob
-
- getHandler() - Method in interface ai.preferred.venom.job.Job
-
Get the handler to handle the response of the job.
- getHandler(Request) - Method in class ai.preferred.venom.UrlRouter
-
- getHeaders() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getHeaders() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getHeaders() - Method in interface ai.preferred.venom.request.Request
-
Returns the headers set for the request.
- getHeaders() - Method in class ai.preferred.venom.request.VRequest
-
- getHeaders() - Method in class ai.preferred.venom.response.BaseResponse
-
- getHeaders() - Method in interface ai.preferred.venom.response.Response
-
Returns the headers that were used to trigger this response.
- getHeaders() - Method in class ai.preferred.venom.response.VResponse
-
- getHtml() - Method in class ai.preferred.venom.response.VResponse
-
Returns the html in string format.
- getHtml(Charset) - Method in class ai.preferred.venom.response.VResponse
-
Returns the html in string format.
- getId() - Method in interface ai.preferred.venom.storage.Record
-
- getInner() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getInner() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getInner() - Method in interface ai.preferred.venom.request.Unwrappable
-
Returns the unwrapped version of this request.
- getInner() - Method in interface ai.preferred.venom.response.Unwrappable
-
Returns the unwrapped version of this request.
- getInner() - Method in class ai.preferred.venom.response.VResponse
-
- getJsoup() - Method in class ai.preferred.venom.response.VResponse
-
Returns a Jsoup document of this response.
- getJsoup(Charset) - Method in class ai.preferred.venom.response.VResponse
-
Returns a Jsoup document of this response.
- getMD5() - Method in interface ai.preferred.venom.storage.Record
-
- getMethod() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getMethod() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getMethod() - Method in interface ai.preferred.venom.request.Request
-
Returns the method type of the request.
- getMethod() - Method in class ai.preferred.venom.request.VRequest
-
- getPriority() - Method in class ai.preferred.venom.job.BasicJob
-
- getPriority() - Method in interface ai.preferred.venom.job.Job
-
Get the current priority set for this job.
- getPriority() - Method in enum ai.preferred.venom.job.Priority
-
Get the current priority level.
- getProxy() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getProxy() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getProxy() - Method in interface ai.preferred.venom.request.Request
-
Returns the proxy set to be used for the request or default to
fetcher if none specified.
- getProxy() - Method in class ai.preferred.venom.request.VRequest
-
- getProxy() - Method in class ai.preferred.venom.response.BaseResponse
-
- getProxy() - Method in interface ai.preferred.venom.response.Response
-
Returns the proxy that was used to trigger this response.
- getProxy() - Method in class ai.preferred.venom.response.VResponse
-
- getProxyList() - Method in interface ai.preferred.venom.ProxyProvider
-
Returns a list of all proxies.
- getRequest() - Method in class ai.preferred.venom.job.BasicJob
-
- getRequest() - Method in interface ai.preferred.venom.job.Job
-
Get the request of this job.
- getRequestBody() - Method in interface ai.preferred.venom.storage.Record
-
- getRequestHeaders() - Method in interface ai.preferred.venom.storage.Record
-
- getRequestMethod() - Method in interface ai.preferred.venom.storage.Record
-
- getResponse() - Method in exception ai.preferred.venom.fetcher.ValidationException
-
Get the response validated.
- getResponseContent() - Method in interface ai.preferred.venom.storage.Record
-
- getResponseHeaders() - Method in interface ai.preferred.venom.storage.Record
-
- getScheduler() - Method in class ai.preferred.venom.Crawler
-
Get the instance of scheduler used.
- getSleepScheduler() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getSleepScheduler() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getSleepScheduler() - Method in interface ai.preferred.venom.request.Request
-
Returns information about the amount of sleep before this request
is made.
- getSleepScheduler() - Method in class ai.preferred.venom.request.VRequest
-
- getSleepTime() - Method in class ai.preferred.venom.SleepScheduler
-
Get the amount of time to wait specified in this class.
- getSourceId() - Method in class ai.preferred.venom.response.BaseResponse
-
- getSourceId() - Method in interface ai.preferred.venom.response.Retrievable
-
Returns the id of the row where an archive of this response
has been insert into a persistent storage.
- getStatus() - Method in exception ai.preferred.venom.fetcher.ValidationException
-
Get the validation status of the response.
- getStatusCode() - Method in class ai.preferred.venom.response.BaseResponse
-
- getStatusCode() - Method in interface ai.preferred.venom.response.Response
-
Returns status code of the response.
- getStatusCode() - Method in class ai.preferred.venom.response.VResponse
-
- getStatusCode() - Method in interface ai.preferred.venom.storage.Record
-
- getStreamResponseContent() - Method in interface ai.preferred.venom.storage.Record
-
- getTryCount() - Method in class ai.preferred.venom.job.BasicJob
-
- getTryCount() - Method in interface ai.preferred.venom.job.Job
-
Get attempt number of this job.
- getUrl() - Method in class ai.preferred.venom.request.CrawlerRequest
-
- getUrl() - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
- getUrl() - Method in interface ai.preferred.venom.request.Request
-
Returns the url of the request.
- getUrl() - Method in class ai.preferred.venom.request.VRequest
-
- getURL() - Method in interface ai.preferred.venom.storage.Record
-
- getValidator() - Method in class ai.preferred.venom.response.BaseResponse
-
- getValidator() - Method in interface ai.preferred.venom.response.Response
-
Returns the instance of validator used to validate this response.
- getValidator() - Method in class ai.preferred.venom.response.VResponse
-
- getValidator(Request) - Method in class ai.preferred.venom.UrlRouter
-
- getValidator(Request) - Method in interface ai.preferred.venom.ValidatorRouter
-
Returns the handler to be used for a specified request.
- getWorker() - Method in class ai.preferred.venom.ThreadedWorkerManager
-
- getWorker() - Method in interface ai.preferred.venom.WorkerManager
-
Get the result collector in use.
- parallism(int) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets the parallelism level.
- peek() - Method in class ai.preferred.venom.job.AbstractQueueScheduler
-
- PipelineValidator - Class in ai.preferred.venom.validator
-
- PipelineValidator(Validator...) - Constructor for class ai.preferred.venom.validator.PipelineValidator
-
Constructs pipeline validator.
- PipelineValidator(List<Validator>) - Constructor for class ai.preferred.venom.validator.PipelineValidator
-
Constructs pipeline validator.
- poll(long, TimeUnit) - Method in class ai.preferred.venom.job.AbstractQueueScheduler
-
- poll() - Method in class ai.preferred.venom.job.AbstractQueueScheduler
-
- poll() - Method in class ai.preferred.venom.job.LazyScheduler
-
- poll(long, TimeUnit) - Method in class ai.preferred.venom.job.LazyScheduler
-
- post(String) - Static method in class ai.preferred.venom.request.VRequest.Builder
-
Creates a new instance of builder with method type post.
- prependHeaders(Map<String, String>) - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
Prepend headers to the current headers.
- Priority - Enum in ai.preferred.venom.job
-
Job priorities, list in descending order of priority:
HIGHEST, HIGH, NORMAL, LOW, LOWEST.
- PriorityQueueScheduler - Class in ai.preferred.venom.job
-
This class provides and implementation of scheduler with a priority
sensitive queue.
- PriorityQueueScheduler() - Constructor for class ai.preferred.venom.job.PriorityQueueScheduler
-
- propRetainProxy(double) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets the proportion of max tries where a specified proxy, if specified will be used.
- proxyProvider(ProxyProvider) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Sets the ProxyProvider to be used.
- ProxyProvider - Interface in ai.preferred.venom
-
This interface allows the user to define proxies to be used for requests.
- put(Job) - Method in class ai.preferred.venom.job.FIFOScheduler
-
- put(Job) - Method in class ai.preferred.venom.job.LazyScheduler
-
- put(Job) - Method in class ai.preferred.venom.job.PriorityQueueScheduler
-
- put(String) - Static method in class ai.preferred.venom.request.VRequest.Builder
-
Creates a new instance of builder with method type put.
- put(Session.Key<T>, T) - Method in class ai.preferred.venom.Session.Builder
-
Adds a session variable into store.
- put(Request, Response) - Method in interface ai.preferred.venom.storage.FileManager
-
Puts record into database.
- Record - Interface in ai.preferred.venom.storage
-
- register(Callback) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Register any callbacks that will be called when a page has been fetched.
- register(Pattern, Handler) - Method in class ai.preferred.venom.UrlRouter
-
Adds a url pattern, and the handler to be used.
- register(Pattern, Validator) - Method in class ai.preferred.venom.UrlRouter
-
Adds a url pattern, and the handler to be used.
- register(Pattern, Handler, Validator) - Method in class ai.preferred.venom.UrlRouter
-
Adds a url pattern, and the handler to be used.
- releaseResources() - Method in class ai.preferred.venom.fetcher.AsyncResponseConsumer
-
- remainingCapacity() - Method in class ai.preferred.venom.job.AbstractQueueScheduler
-
- remove(HttpHost) - Method in interface ai.preferred.venom.ProxyProvider
-
Remove a proxy from the list.
- removeHeader(String) - Method in class ai.preferred.venom.request.VRequest.Builder
-
Remove a header from this request.
- removeHeaders() - Method in class ai.preferred.venom.request.VRequest.Builder
-
Remove all headers from this request.
- removeProxy() - Method in class ai.preferred.venom.request.CrawlerRequest
-
Remove the proxy from this request.
- Request - Interface in ai.preferred.venom.request
-
- Request.Method - Enum in ai.preferred.venom.request
-
The method of the request to be made.
- reQueue() - Method in class ai.preferred.venom.job.BasicJob
-
- reQueue() - Method in interface ai.preferred.venom.job.Job
-
Remove any existing in queue, downgrades the priority and
adds the job back into queue.
- Response - Interface in ai.preferred.venom.response
-
- ResponseDecompressor - Class in ai.preferred.venom.utils
-
Modified from: org.apache.http.client.protocol.ResponseContentEncoding.
- ResponseDecompressor() - Constructor for class ai.preferred.venom.utils.ResponseDecompressor
-
Constructs a decompressor.
- Retrievable - Interface in ai.preferred.venom.response
-
This interface represents that the response can be/ has been stored.
- router(HandlerRouter) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets HandlerRouter to be used.
- router(ValidatorRouter) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Sets ValidatorRouter to be used.
- scheduler(AbstractQueueScheduler) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets the Scheduler to be used, if not set, default will be chosen.
- Scheduler - Interface in ai.preferred.venom.job
-
This interface represents only the most basic of a scheduler.
- session(Session) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets the Session to be used, if not set, defaults to none.
- Session - Class in ai.preferred.venom
-
- Session.Builder - Class in ai.preferred.venom
-
Builder for Session.
- Session.Key<T> - Class in ai.preferred.venom
-
A class representing the key for a session.
- setBody(String) - Method in class ai.preferred.venom.request.VRequest.Builder
-
Sets the request body to be used.
- setProxy(HttpHost) - Method in class ai.preferred.venom.request.HttpFetcherRequest
-
Sets proxy to be used for this request.
- setProxy(HttpHost) - Method in class ai.preferred.venom.request.VRequest.Builder
-
Sets the proxy to be used, this will override the
proxy selected in Fetcher for this request.
- setSleepScheduler(SleepScheduler) - Method in class ai.preferred.venom.request.VRequest.Builder
-
Sets the sleep scheduler to be used, this will override the
sleep scheduler defined in Crawler for this request.
- setSourceId(String) - Method in class ai.preferred.venom.response.BaseResponse
-
Sets the source id where the raw response is saved.
- setUrl(String) - Method in class ai.preferred.venom.request.VRequest.Builder
-
Sets the url to be fetched.
- setValidator(Validator) - Method in class ai.preferred.venom.response.BaseResponse
-
Sets the validator used to validate this response.
- shutdown() - Method in class ai.preferred.venom.utils.InlineExecutorService
-
- shutdownNow() - Method in class ai.preferred.venom.utils.InlineExecutorService
-
- size() - Method in class ai.preferred.venom.job.AbstractQueueScheduler
-
- sleepScheduler(SleepScheduler) - Method in class ai.preferred.venom.Crawler.Builder
-
Sets the SleepScheduler to be used, if not set, default will be chosen.
- SleepScheduler - Class in ai.preferred.venom
-
- SleepScheduler(long) - Constructor for class ai.preferred.venom.SleepScheduler
-
Constructs a sleep scheduler with fix sleep time.
- SleepScheduler(long, long) - Constructor for class ai.preferred.venom.SleepScheduler
-
Constructs a sleep scheduler with range of sleep time.
- socketTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Defines the socket timeout (SO_TIMEOUT
) in milliseconds,
which is the timeout for waiting for data or, put differently,
a maximum period inactivity between two consecutive data packets).
- soTimeout(int) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Determines the default socket timeout value for non-blocking I/O operations.
- start() - Method in class ai.preferred.venom.Crawler
-
Starts the crawler by starting a new thread to poll for jobs.
- start() - Method in class ai.preferred.venom.fetcher.AsyncFetcher
-
- start() - Method in interface ai.preferred.venom.fetcher.Fetcher
-
Fetcher starter.
- startAndClose() - Method in class ai.preferred.venom.Crawler
-
Starts the crawler by starting a new thread to poll for jobs and close it
after the queue has reached 0.
- StatusOkValidator - Class in ai.preferred.venom.validator
-
This class determines the validity of a response by its status code.
- StatusOkValidator() - Constructor for class ai.preferred.venom.validator.StatusOkValidator
-
- StopCodeException - Exception in ai.preferred.venom.fetcher
-
- StopCodeException(int, String) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException
-
Constructs a stop code exception with a message.
- StopCodeException(int, String, Throwable) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException
-
Constructs a stop code exception with a message and a cause.
- StopCodeException(int, Throwable) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException
-
Constructs a stop code exception with a cause.
- StopCodeException(int, String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.fetcher.StopCodeException
-
Constructs a stop code exception with a message and a cause.
- stopCodes(int...) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Set a list of stop code that will interrupt crawling.
- StorageException - Exception in ai.preferred.venom.storage
-
- StorageException(String) - Constructor for exception ai.preferred.venom.storage.StorageException
-
Constructs a storage exception with a message.
- StorageException(String, Throwable) - Constructor for exception ai.preferred.venom.storage.StorageException
-
Constructs a storage exception with a message and a cause.
- StorageException(Throwable) - Constructor for exception ai.preferred.venom.storage.StorageException
-
Constructs a storage exception with a cause.
- StorageException(String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.storage.StorageException
-
Constructs a storage exception with a message and a cause.
- submit(Callable<T>) - Method in interface ai.preferred.venom.Worker
-
Submits a value-returning task for execution and returns a
Future representing the pending results of the task.
- submit(Runnable, T) - Method in interface ai.preferred.venom.Worker
-
Submits a Runnable task for execution and returns a Future
representing that task.
- submit(Runnable) - Method in interface ai.preferred.venom.Worker
-
Submits a Runnable task for execution and returns a Future
representing that task.
- ValidationException - Exception in ai.preferred.venom.fetcher
-
- ValidationException(Validator.Status, Response, String) - Constructor for exception ai.preferred.venom.fetcher.ValidationException
-
Constructs a validation exception with a message.
- ValidationException(Validator.Status, Response, String, Throwable) - Constructor for exception ai.preferred.venom.fetcher.ValidationException
-
- ValidationException(Validator.Status, Response, Throwable) - Constructor for exception ai.preferred.venom.fetcher.ValidationException
-
- ValidationException(Validator.Status, Response, String, Throwable, boolean, boolean) - Constructor for exception ai.preferred.venom.fetcher.ValidationException
-
- validator(Validator) - Method in class ai.preferred.venom.fetcher.AsyncFetcher.Builder
-
Sets the Validator to be used.
- Validator - Interface in ai.preferred.venom.validator
-
- Validator.Status - Enum in ai.preferred.venom.validator
-
The allowed return status of validation.
- ValidatorRouter - Interface in ai.preferred.venom
-
This interface allows the user to map request to handleable.
- valueOf(String) - Static method in enum ai.preferred.venom.job.Priority
-
Returns the enum constant of this type with the specified name.
- valueOf(String) - Static method in enum ai.preferred.venom.request.Request.Method
-
Returns the enum constant of this type with the specified name.
- valueOf(String) - Static method in enum ai.preferred.venom.validator.Validator.Status
-
Returns the enum constant of this type with the specified name.
- values() - Static method in enum ai.preferred.venom.job.Priority
-
Returns an array containing the constants of this enum type, in
the order they are declared.
- values() - Static method in enum ai.preferred.venom.request.Request.Method
-
Returns an array containing the constants of this enum type, in
the order they are declared.
- values() - Static method in enum ai.preferred.venom.validator.Validator.Status
-
Returns an array containing the constants of this enum type, in
the order they are declared.
- VRequest - Class in ai.preferred.venom.request
-
An implementation of HTTP request.
- VRequest(String) - Constructor for class ai.preferred.venom.request.VRequest
-
Constructs an instance of venom request.
- VRequest(String, Map<String, String>) - Constructor for class ai.preferred.venom.request.VRequest
-
Constructs an instance of venom request.
- VRequest(VRequest.Builder<?>) - Constructor for class ai.preferred.venom.request.VRequest
-
Constructs an instance of venom request.
- VRequest.Builder<T extends VRequest.Builder<T>> - Class in ai.preferred.venom.request
-
A builder for VRequest class.
- VResponse - Class in ai.preferred.venom.response
-
- VResponse(Response) - Constructor for class ai.preferred.venom.response.VResponse
-
Constructs a VResponse.