Package ai.preferred.venom
Class Crawler
- java.lang.Object
-
- ai.preferred.venom.Crawler
-
- All Implemented Interfaces:
Interruptible
,java.lang.AutoCloseable
public final class Crawler extends java.lang.Object implements Interruptible
This class handles the coordination between classes during the pre and post fetching of a page such as executing threads, calling to fetcher and manipulating the priority of a scheduled request.- Author:
- Maksim Tkachenko, Truong Quoc Tuan, Ween Jiann Lee
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
Crawler.AsyncCrawlerCallbackProcessor
This class methods is executed upon the completion of fetcher.static class
Crawler.Builder
A builder for crawler class.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static Crawler
buildDefault()
Builds a new default instance of Crawler.static Crawler.Builder
builder()
Creates a new instance of Builder.void
close()
Scheduler
getScheduler()
Get the instance of scheduler used.void
interruptAndClose()
Interrupt a thread and then close it.Crawler
start()
Starts the crawler by starting a new thread to poll for jobs.Crawler
startAndClose()
Starts the crawler by starting a new thread to poll for jobs and close it after the queue has reached 0.
-
-
-
Method Detail
-
builder
public static Crawler.Builder builder()
Creates a new instance of Builder.- Returns:
- an instance of Builder.
-
buildDefault
public static Crawler buildDefault()
Builds a new default instance of Crawler.- Returns:
- an instance of Crawler.
-
getScheduler
public Scheduler getScheduler()
Get the instance of scheduler used.- Returns:
- the instance of scheduler used.
-
start
public Crawler start()
Starts the crawler by starting a new thread to poll for jobs.- Returns:
- the instance of Crawler used.
-
startAndClose
public Crawler startAndClose() throws java.lang.Exception
Starts the crawler by starting a new thread to poll for jobs and close it after the queue has reached 0.- Returns:
- the instance of Crawler used.
- Throws:
java.lang.Exception
- if this resource cannot be closed.
-
interruptAndClose
public void interruptAndClose() throws java.lang.Exception
Description copied from interface:Interruptible
Interrupt a thread and then close it.- Specified by:
interruptAndClose
in interfaceInterruptible
- Throws:
java.lang.Exception
- Exception.
-
close
public void close() throws java.lang.Exception
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Throws:
java.lang.Exception
-
-