ai.preferred.venom

Class Crawler

java.lang.Object

ai.preferred.venom.Crawler

All Implemented Interfaces:

Interruptible, AutoCloseable

public final class Crawler
extends Object
implements Interruptible

This class handles the coordination between classes during the pre and post fetching of a page such as executing threads, calling to fetcher and manipulating the priority of a scheduled request.

Author:

Maksim Tkachenko, Truong Quoc Tuan, Ween Jiann Lee

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Crawler.AsyncCrawlerCallbackProcessor This class methods is executed upon the completion of fetcher.
static class	Crawler.Builder A builder for crawler class.

Method Summary

Modifier and Type	Method and Description
static Crawler	buildDefault() Builds a new default instance of Crawler.
static Crawler.Builder	builder() Creates a new instance of Builder.
void	close()
Scheduler	getScheduler() Get the instance of scheduler used.
void	interruptAndClose() Interrupt a thread and then close it.
Crawler	start() Starts the crawler by starting a new thread to poll for jobs.
Crawler	startAndClose() Starts the crawler by starting a new thread to poll for jobs and close it after the queue has reached 0.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static Crawler.Builder builder()

Creates a new instance of Builder.

Returns:

an instance of Builder.

buildDefault

public static Crawler buildDefault()

Builds a new default instance of Crawler.

Returns:

an instance of Crawler.

getScheduler

public Scheduler getScheduler()

Get the instance of scheduler used.

Returns:

the instance of scheduler used.

start

public Crawler start()

Starts the crawler by starting a new thread to poll for jobs.

Returns:

the instance of Crawler used.

startAndClose

public Crawler startAndClose()
                      throws Exception

Starts the crawler by starting a new thread to poll for jobs and close it after the queue has reached 0.

Returns:

the instance of Crawler used.

Throws:

Exception - if this resource cannot be closed.

interruptAndClose

public void interruptAndClose()
                       throws Exception

Description copied from interface: Interruptible

Interrupt a thread and then close it.

Specified by:

interruptAndClose in interface Interruptible

Throws:

Exception - Exception.

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception