ai.preferred.venom

Class Crawler.Builder

java.lang.Object

ai.preferred.venom.Crawler.Builder

Enclosing class:

Crawler

public static final class Crawler.Builder
extends Object

A builder for crawler class.

Method Summary

Modifier and Type	Method and Description
Crawler	build() Builds the crawler with the options specified.
Crawler.Builder	fetcher(@NotNull Fetcher fetcher) Sets the Fetcher to be used, if not set, default will be chosen.
Crawler.Builder	maxConnections(int maxConnections) The number of concurrent connections allowed out of the client.
Crawler.Builder	maxTries(int maxTries) Sets number of times to retry for a request.
Crawler.Builder	name(@NotNull String name) Sets the name for crawler thread.
Crawler.Builder	parallism(int parallelism) Sets the parallelism level.
Crawler.Builder	propRetainProxy(double propRetainProxy) Sets the proportion of max tries where a specified proxy, if specified will be used.
Crawler.Builder	router(@NotNull HandlerRouter router) Sets HandlerRouter to be used.
Crawler.Builder	scheduler(@NotNull AbstractQueueScheduler scheduler) Sets the Scheduler to be used, if not set, default will be chosen.
Crawler.Builder	session(@NotNull Session session) Sets the Session to be used, if not set, defaults to none.
Crawler.Builder	sleepScheduler(@NotNull SleepScheduler sleepScheduler) Sets the SleepScheduler to be used, if not set, default will be chosen.
Crawler.Builder	workerManager(@NotNull WorkerManager workerManager) Sets the WorkerManager to be used, if not set, default will be chosen.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

name

public Crawler.Builder name(@NotNull
                            @NotNull String name)

Sets the name for crawler thread.

Parameters:

name - name for crawler thread

Returns:

this

fetcher

public Crawler.Builder fetcher(@NotNull
                               @NotNull Fetcher fetcher)

Sets the Fetcher to be used, if not set, default will be chosen.

Parameters:

fetcher - fetcher to be used.

Returns:

this

parallism

public Crawler.Builder parallism(int parallelism)

Sets the parallelism level. Defaults to system thread count.

Parameters:

parallelism - the parallelism level.

Returns:

this

workerManager

public Crawler.Builder workerManager(@NotNull
                                     @NotNull WorkerManager workerManager)

Sets the WorkerManager to be used, if not set, default will be chosen.

Parameters:

workerManager - result workerManager to be used.

Returns:

this

scheduler

public Crawler.Builder scheduler(@NotNull
                                 @NotNull AbstractQueueScheduler scheduler)

Sets the Scheduler to be used, if not set, default will be chosen.

Parameters:

scheduler - scheduler to be used.

Returns:

this

router

public Crawler.Builder router(@NotNull
                              @NotNull HandlerRouter router)

Sets HandlerRouter to be used. Defaults to none.

Parameters:

router - handler router to be used.

Returns:

this

maxConnections

public Crawler.Builder maxConnections(int maxConnections)

The number of concurrent connections allowed out of the client.

Parameters:

maxConnections - maximum number of concurrent connections.

Returns:

this

maxTries

public Crawler.Builder maxTries(int maxTries)

Sets number of times to retry for a request. This number excludes the first try. Defaults to 50.

Parameters:

maxTries - max retry times.

Returns:

this

propRetainProxy

public Crawler.Builder propRetainProxy(double propRetainProxy)

Sets the proportion of max tries where a specified proxy, if specified will be used. Number should be between 0 and 1 inclusive, Defaults to 0.05.

This only comes into effect when a specific proxy is set for the request. This proxy set will be overridden beyond this threshold.

Parameters:

propRetainProxy - threshold percentage.

Returns:

this

sleepScheduler

public Crawler.Builder sleepScheduler(@NotNull
                                      @NotNull SleepScheduler sleepScheduler)

Sets the SleepScheduler to be used, if not set, default will be chosen.

Parameters:

sleepScheduler - sleepAndGetTime scheduler to be used.

Returns:

this

session

public Crawler.Builder session(@NotNull
                               @NotNull Session session)

Sets the Session to be used, if not set, defaults to none.

Parameters:

session - Sessions where variables are defined

Returns:

this

build

public Crawler build()

Builds the crawler with the options specified.

Returns:

an instance of Crawler