class PyPiReleasesSource extends RichSourceFunction[PyPiRelease] with CheckpointedFunction
- Alphabetic
- By Inheritance
- PyPiReleasesSource
- CheckpointedFunction
- RichSourceFunction
- SourceFunction
- AbstractRichFunction
- RichFunction
- Function
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new PyPiReleasesSource(config: PyPiSourceConfig = PyPiSourceConfig())
Value Members
-
def
cancel(): Unit
Close the source.
Close the source.
- Definition Classes
- PyPiReleasesSource → SourceFunction
-
def
close(): Unit
- Definition Classes
- AbstractRichFunction → RichFunction
- Annotations
- @throws( classOf[java.lang.Exception] )
-
val
dateFormat: String
Format and URL of RSS Feed.
-
def
decreaseRunsLeft(): Unit
If there is a limit to the amount of runs decrease by 1
- def getIsRunning: Boolean
-
def
getIterationRuntimeContext(): IterationRuntimeContext
- Definition Classes
- AbstractRichFunction → RichFunction
-
def
getRSSAsString: String
Requests the RSS feed and returns its body as a string.
Requests the RSS feed and returns its body as a string. Will keep trying with increasing intervals if it doesn't succeed
- returns
Body of requested RSS feed
- Annotations
- @throws( ... )
-
def
getRuntimeContext(): RuntimeContext
- Definition Classes
- AbstractRichFunction → RichFunction
-
def
initializeState(context: FunctionInitializationContext): Unit
Initializes state by reading from a checkpoint or creating an empty one.
Initializes state by reading from a checkpoint or creating an empty one.
- Definition Classes
- PyPiReleasesSource → CheckpointedFunction
-
def
open(parameters: Configuration): Unit
Opens this source.
Opens this source.
- Definition Classes
- PyPiReleasesSource → AbstractRichFunction → RichFunction
-
def
parseRSSString(rssString: String): Seq[PyPiRelease]
Parses a string that contains xml with RSS items
Parses a string that contains xml with RSS items
- rssString
XML string with RSS items
- returns
Sequence of RSS items
-
val
releasesProcessed: LongCounter
Accumulator for the amount of processed releases.
-
def
run(ctx: SourceContext[PyPiRelease]): Unit
Runs the source.
-
def
setRuntimeContext(arg0: RuntimeContext): Unit
- Definition Classes
- AbstractRichFunction → RichFunction
-
def
snapshotState(context: FunctionSnapshotContext): Unit
Make a snapshot of the current state.
Make a snapshot of the current state.
- Definition Classes
- PyPiReleasesSource → CheckpointedFunction
-
def
sortAndDropDuplicates(items: Seq[PyPiRelease]): Seq[PyPiRelease]
Drops items that already have been collected and sorts them based on times
Drops items that already have been collected and sorts them based on times
- items
Potential items to be collected
- returns
Valid sorted items
- val url: String
-
def
waitPollingInterval(times: Int = 1): Unit
Wait a certain amount of times the polling interval
Wait a certain amount of times the polling interval
- times
Times the polling interval should be waited
-
def
xmlToPyPiRelease(node: Node): PyPiRelease
Parses a xml node to a RSS item
Parses a xml node to a RSS item
- node
XML node
- returns
RSS item