Package io.pravega.connectors.flink

Class AbstractStreamingReaderBuilder<T,B extends AbstractStreamingReaderBuilder>

java.lang.Object

io.pravega.connectors.flink.AbstractReaderBuilder<B>

io.pravega.connectors.flink.AbstractStreamingReaderBuilder<T,B>

Type Parameters:

T - the element type.

B - the builder type.

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

FlinkPravegaReader.Builder, Pravega.TableSourceReaderBuilder

@Internal
public abstract class AbstractStreamingReaderBuilder<T,B extends AbstractStreamingReaderBuilder>
extends AbstractReaderBuilder<B>

An abstract streaming reader builder. The builder is abstracted to act as the base for both the FlinkPravegaReader and FlinkPravegaTableSource builders.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	AbstractStreamingReaderBuilder.ReaderGroupInfo

Field Summary

Fields

Modifier and Type	Field	Description
org.apache.flink.api.common.time.Time	checkpointInitiateTimeout
org.apache.flink.api.common.time.Time	eventReadTimeout
int	maxOutstandingCheckpointRequest
java.lang.String	readerGroupName
org.apache.flink.api.common.time.Time	readerGroupRefreshTime
java.lang.String	readerGroupScope
java.lang.String	uid

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	AbstractStreamingReaderBuilder()

Method Summary

Modifier and Type	Method	Description
AbstractStreamingReaderBuilder.ReaderGroupInfo	buildReaderGroupInfo()	Build reader group configuration.
protected FlinkPravegaReader<T>	buildSourceFunction()	Builds a FlinkPravegaReader based on the configuration.
java.lang.String	generateUid()	Generate a UID for the source, to distinguish the state associated with the checkpoint hook.
protected abstract org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>>	getAssignerWithTimeWindows()
protected abstract org.apache.flink.api.common.serialization.DeserializationSchema<T>	getDeserializationSchema()
B	uid(java.lang.String uid)	Configures the source uid to identify the checkpoint state of this source.
B	withCheckpointInitiateTimeout(org.apache.flink.api.common.time.Time checkpointInitiateTimeout)	Sets the timeout for initiating a checkpoint in Pravega.
B	withEventReadTimeout(org.apache.flink.api.common.time.Time eventReadTimeout)	Sets the timeout for the call to read events from Pravega.
B	withMaxOutstandingCheckpointRequest(int maxOutstandingCheckpointRequest)	Configures the maximum outstanding checkpoint requests to Pravega (default=3).
B	withReaderGroupName(java.lang.String readerGroupName)	Configures the reader group name.
B	withReaderGroupRefreshTime(org.apache.flink.api.common.time.Time groupRefreshTime)	Sets the group refresh time, with a default of 1 second.
B	withReaderGroupScope(java.lang.String scope)	Configures the reader group scope for synchronization purposes.

Methods inherited from class io.pravega.connectors.flink.AbstractReaderBuilder

builder, enableMetrics, forStream, forStream, forStream, forStream, forStream, forStream, getPravegaConfig, isMetricsEnabled, resolveStreams, withPravegaConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

uid

public java.lang.String uid

readerGroupScope

public java.lang.String readerGroupScope

readerGroupName

public java.lang.String readerGroupName

readerGroupRefreshTime

public org.apache.flink.api.common.time.Time readerGroupRefreshTime

checkpointInitiateTimeout

public org.apache.flink.api.common.time.Time checkpointInitiateTimeout

eventReadTimeout

public org.apache.flink.api.common.time.Time eventReadTimeout

maxOutstandingCheckpointRequest

public int maxOutstandingCheckpointRequest

Constructor Detail

AbstractStreamingReaderBuilder

protected AbstractStreamingReaderBuilder()

Method Detail

uid

public B uid(java.lang.String uid)

Configures the source uid to identify the checkpoint state of this source.

The default value is generated based on other inputs.

Parameters:

uid - the uid to use.

Returns:

A builder to configure and create a streaming reader.

withReaderGroupScope

public B withReaderGroupScope(java.lang.String scope)

Configures the reader group scope for synchronization purposes.

The default value is taken from the PravegaConfig defaultScope property.

Parameters:

scope - the scope name.

Returns:

A builder to configure and create a streaming reader.

withReaderGroupName

public B withReaderGroupName(java.lang.String readerGroupName)

Configures the reader group name.

Parameters:

readerGroupName - the reader group name.

Returns:

A builder to configure and create a streaming reader.

withReaderGroupRefreshTime

public B withReaderGroupRefreshTime(org.apache.flink.api.common.time.Time groupRefreshTime)

Sets the group refresh time, with a default of 1 second.

Parameters:

groupRefreshTime - The group refresh time

Returns:

A builder to configure and create a streaming reader.

withCheckpointInitiateTimeout

public B withCheckpointInitiateTimeout(org.apache.flink.api.common.time.Time checkpointInitiateTimeout)

Sets the timeout for initiating a checkpoint in Pravega.

Parameters:

checkpointInitiateTimeout - The timeout

Returns:

A builder to configure and create a streaming reader.

withEventReadTimeout

public B withEventReadTimeout(org.apache.flink.api.common.time.Time eventReadTimeout)

Sets the timeout for the call to read events from Pravega. After the timeout expires (without an event being returned), another call will be made.

Parameters:

eventReadTimeout - The timeout

Returns:

A builder to configure and create a streaming reader.

withMaxOutstandingCheckpointRequest

public B withMaxOutstandingCheckpointRequest(int maxOutstandingCheckpointRequest)

Configures the maximum outstanding checkpoint requests to Pravega (default=3). Upon requesting more checkpoints than the specified maximum, (say a checkpoint request times out on the ReaderCheckpointHook but Pravega is still working on it), this configurations allows Pravega to limit any further checkpoint request being made to the ReaderGroup. This configuration is particularly relevant when multiple checkpoint requests need to be honored (e.g., frequent savepoint requests being triggered concurrently).

Parameters:

maxOutstandingCheckpointRequest - maximum outstanding checkpoint request.

Returns:

A builder to configure and create a streaming reader.

getDeserializationSchema

protected abstract org.apache.flink.api.common.serialization.DeserializationSchema<T> getDeserializationSchema()

getAssignerWithTimeWindows

protected abstract org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>> getAssignerWithTimeWindows()

buildSourceFunction

protected FlinkPravegaReader<T> buildSourceFunction()

Builds a FlinkPravegaReader based on the configuration. Note that the FlinkPravegaTableSource supports both the batch and streaming API, and so creates both a source function and an input format and then uses one or the other. Be sure to call initialize() before returning the reader to user code.

Returns:

an uninitiailized reader as a source function.

Throws:

java.lang.IllegalStateException - if the configuration is invalid.

buildReaderGroupInfo

public AbstractStreamingReaderBuilder.ReaderGroupInfo buildReaderGroupInfo()

Build reader group configuration.

Returns:

AbstractStreamingReaderBuilder.ReaderGroupInfo

generateUid

public java.lang.String generateUid()

Generate a UID for the source, to distinguish the state associated with the checkpoint hook. A good generated UID will: 1. be stable across savepoints for the same inputs 2. disambiguate one source from another (e.g. in a program that uses numerous instances of FlinkPravegaReader) 3. allow for reconfiguration of the stream cuts and timeouts

Returns:

A generated reader group ID.