Package | Description |
---|---|
ai.preferred.venom | |
ai.preferred.venom.fetcher | |
ai.preferred.venom.job | |
ai.preferred.venom.request | |
ai.preferred.venom.storage | |
ai.preferred.venom.validator |
Modifier and Type | Method and Description |
---|---|
void |
Crawler.AsyncCrawlerCallbackProcessor.cancelled(Request request) |
void |
Crawler.AsyncCrawlerCallbackProcessor.completed(Request request,
Response response) |
void |
Crawler.AsyncCrawlerCallbackProcessor.failed(Request request,
Exception ex) |
org.apache.http.HttpHost |
ProxyProvider.get(@NotNull Request request)
Returns the get proxy from the list.
|
Handler |
UrlRouter.getHandler(Request request) |
Handler |
HandlerRouter.getHandler(@NotNull Request request)
Returns the handler to be used for a specified request.
|
Validator |
ValidatorRouter.getValidator(@NotNull Request request)
Returns the handler to be used for a specified request.
|
Validator |
UrlRouter.getValidator(Request request) |
void |
Handler.handle(Request request,
VResponse response,
Scheduler scheduler,
Session session,
Worker worker)
This function is called when the request is fetched successfully.
|
Modifier and Type | Method and Description |
---|---|
void |
Callback.cancelled(@NotNull Request request)
Method to be call upon cancellation of request.
|
void |
Callback.completed(@NotNull Request request,
@NotNull Response response)
Method to be call upon completion of request.
|
void |
Callback.failed(@NotNull Request request,
@NotNull Exception ex)
Method to be call upon failure of request.
|
@NotNull Future<Response> |
Fetcher.fetch(@NotNull Request request)
Fetch the desired HTTP page given in
Request . |
Future<Response> |
AsyncFetcher.fetch(Request request) |
@NotNull Future<Response> |
Fetcher.fetch(@NotNull Request request,
@NotNull Callback callback)
Fetch the desired HTTP page given in
Request . |
Future<Response> |
AsyncFetcher.fetch(Request request,
Callback callback) |
Modifier and Type | Method and Description |
---|---|
@NotNull Request |
Job.getRequest()
Get the request of this job.
|
Request |
BasicJob.getRequest() |
Modifier and Type | Method and Description |
---|---|
void |
Scheduler.add(@NotNull Request r)
Adds a request to the queue.
|
void |
AbstractQueueScheduler.add(Request r) |
void |
Scheduler.add(@NotNull Request r,
@NotNull Handler h)
Adds a request to the queue.
|
void |
AbstractQueueScheduler.add(Request r,
Handler h) |
void |
Scheduler.add(@NotNull Request r,
@NotNull Handler h,
Priority p)
Adds a request to the queue.
|
void |
AbstractQueueScheduler.add(Request r,
Handler h,
Priority p) |
void |
Scheduler.add(@NotNull Request r,
@NotNull Handler h,
Priority p,
Priority pf)
Adds a request to the queue.
|
void |
PriorityQueueScheduler.add(Request r,
Handler h,
Priority p,
Priority pf) |
void |
LazyScheduler.add(Request r,
Handler h,
Priority p,
Priority pf) |
void |
FIFOScheduler.add(@NotNull Request r,
@NotNull Handler h,
Priority p,
Priority pf) |
void |
Scheduler.add(@NotNull Request r,
Priority p)
Adds a request to the queue.
|
void |
AbstractQueueScheduler.add(Request r,
Priority p) |
void |
Scheduler.add(@NotNull Request r,
Priority p,
Priority pf)
Adds a request to the queue.
|
void |
AbstractQueueScheduler.add(Request r,
Priority p,
Priority pf) |
Constructor and Description |
---|
BasicJob(Request request,
Handler handler,
Priority priority,
Priority priorityFloor,
Queue<Job> queue)
Constructs a basic job.
|
Constructor and Description |
---|
LazyScheduler(Iterator<Request> requests)
Constructs an instance of lazy scheduler without a default handler.
|
LazyScheduler(Iterator<Request> requests,
Handler handler)
Constructs an instance of lazy scheduler with a default handler.
|
Modifier and Type | Interface and Description |
---|---|
interface |
Unwrappable
This interface represents that the request can be unwrapped.
|
Modifier and Type | Class and Description |
---|---|
class |
CrawlerRequest
This class allows the removal of proxy from request.
|
class |
HttpFetcherRequest |
class |
VRequest
An implementation of HTTP request.
|
Modifier and Type | Method and Description |
---|---|
@NotNull Request |
Unwrappable.getInner()
Returns the unwrapped version of this request.
|
Request |
HttpFetcherRequest.getInner() |
Request |
CrawlerRequest.getInner() |
static @NotNull Request |
Unwrappable.unwrapRequest(@NotNull Request request)
Unwrap all wrapped request to an instance of base request.
|
Modifier and Type | Method and Description |
---|---|
static @NotNull Request |
Unwrappable.unwrapRequest(@NotNull Request request)
Unwrap all wrapped request to an instance of base request.
|
Constructor and Description |
---|
CrawlerRequest(Request request)
Constructs an instance of crawler request with an underlying
request.
|
HttpFetcherRequest(Request innerRequest)
Constructs an instance of http fetcher request.
|
Modifier and Type | Method and Description |
---|---|
@NotNull Record |
FileManager.get(@NotNull Request request)
Returns latest record matching request.
|
@NotNull String |
FileManager.put(@NotNull Request request,
@NotNull Response response)
Puts record into database.
|
Modifier and Type | Method and Description |
---|---|
Validator.Status |
Validator.isValid(@NotNull Request request,
@NotNull Response response)
Method will be called when a response need validation.
|
Validator.Status |
StatusOkValidator.isValid(Request request,
Response response) |
Validator.Status |
PipelineValidator.isValid(Request request,
Response response) |
Validator.Status |
MimeTypeValidator.isValid(Request request,
Response response) |
Validator.Status |
EmptyContentValidator.isValid(Request request,
Response response) |
Copyright © 2019. All rights reserved.