Package io.pravega.connectors.flink
Class AbstractStreamingReaderBuilder<T,B extends AbstractStreamingReaderBuilder>
- java.lang.Object
-
- io.pravega.connectors.flink.AbstractReaderBuilder<B>
-
- io.pravega.connectors.flink.AbstractStreamingReaderBuilder<T,B>
-
- Type Parameters:
T
- the element type.B
- the builder type.
- All Implemented Interfaces:
java.io.Serializable
- Direct Known Subclasses:
FlinkPravegaReader.Builder
,PravegaSourceBuilder
@Internal public abstract class AbstractStreamingReaderBuilder<T,B extends AbstractStreamingReaderBuilder> extends AbstractReaderBuilder<B>
An abstract streaming reader builder. The builder is abstracted to act as the base for both theFlinkPravegaReader
builders.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
AbstractStreamingReaderBuilder.ReaderGroupInfo
-
Field Summary
Fields Modifier and Type Field Description org.apache.flink.api.common.time.Time
checkpointInitiateTimeout
org.apache.flink.api.common.time.Time
eventReadTimeout
int
maxOutstandingCheckpointRequest
java.lang.String
readerGroupName
org.apache.flink.api.common.time.Time
readerGroupRefreshTime
java.lang.String
readerGroupScope
java.lang.String
uid
-
Constructor Summary
Constructors Modifier Constructor Description protected
AbstractStreamingReaderBuilder()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description AbstractStreamingReaderBuilder.ReaderGroupInfo
buildReaderGroupInfo()
Build reader group configuration.protected FlinkPravegaReader<T>
buildSourceFunction()
Builds aFlinkPravegaReader
based on the configuration.java.lang.String
generateUid()
Generate a UID for the source, to distinguish the state associated with the checkpoint hook.protected abstract org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>>
getAssignerWithTimeWindows()
protected abstract org.apache.flink.api.common.serialization.DeserializationSchema<T>
getDeserializationSchema()
B
uid(java.lang.String uid)
Configures the source uid to identify the checkpoint state of this source.B
withCheckpointInitiateTimeout(org.apache.flink.api.common.time.Time checkpointInitiateTimeout)
Sets the timeout for initiating a checkpoint in Pravega.B
withEventReadTimeout(org.apache.flink.api.common.time.Time eventReadTimeout)
Sets the timeout for the call to read events from Pravega.B
withMaxOutstandingCheckpointRequest(int maxOutstandingCheckpointRequest)
Configures the maximum outstanding checkpoint requests to Pravega (default=3).B
withReaderGroupName(java.lang.String readerGroupName)
Configures the reader group name.B
withReaderGroupRefreshTime(org.apache.flink.api.common.time.Time groupRefreshTime)
Sets the group refresh time, with a default of 1 second.B
withReaderGroupScope(java.lang.String scope)
Configures the reader group scope for synchronization purposes.-
Methods inherited from class io.pravega.connectors.flink.AbstractReaderBuilder
builder, enableMetrics, forStream, forStream, forStream, forStream, forStream, forStream, getPravegaConfig, isMetricsEnabled, resolveStreams, withPravegaConfig
-
-
-
-
Field Detail
-
uid
public java.lang.String uid
-
readerGroupScope
public java.lang.String readerGroupScope
-
readerGroupName
public java.lang.String readerGroupName
-
readerGroupRefreshTime
public org.apache.flink.api.common.time.Time readerGroupRefreshTime
-
checkpointInitiateTimeout
public org.apache.flink.api.common.time.Time checkpointInitiateTimeout
-
eventReadTimeout
public org.apache.flink.api.common.time.Time eventReadTimeout
-
maxOutstandingCheckpointRequest
public int maxOutstandingCheckpointRequest
-
-
Method Detail
-
uid
public B uid(java.lang.String uid)
Configures the source uid to identify the checkpoint state of this source.The default value is generated based on other inputs.
- Parameters:
uid
- the uid to use.- Returns:
- A builder to configure and create a streaming reader.
-
withReaderGroupScope
public B withReaderGroupScope(java.lang.String scope)
Configures the reader group scope for synchronization purposes.The default value is taken from the
PravegaConfig
defaultScope
property.- Parameters:
scope
- the scope name.- Returns:
- A builder to configure and create a streaming reader.
-
withReaderGroupName
public B withReaderGroupName(java.lang.String readerGroupName)
Configures the reader group name.- Parameters:
readerGroupName
- the reader group name.- Returns:
- A builder to configure and create a streaming reader.
-
withReaderGroupRefreshTime
public B withReaderGroupRefreshTime(org.apache.flink.api.common.time.Time groupRefreshTime)
Sets the group refresh time, with a default of 1 second.- Parameters:
groupRefreshTime
- The group refresh time- Returns:
- A builder to configure and create a streaming reader.
-
withCheckpointInitiateTimeout
public B withCheckpointInitiateTimeout(org.apache.flink.api.common.time.Time checkpointInitiateTimeout)
Sets the timeout for initiating a checkpoint in Pravega.- Parameters:
checkpointInitiateTimeout
- The timeout- Returns:
- A builder to configure and create a streaming reader.
-
withEventReadTimeout
public B withEventReadTimeout(org.apache.flink.api.common.time.Time eventReadTimeout)
Sets the timeout for the call to read events from Pravega. After the timeout expires (without an event being returned), another call will be made.- Parameters:
eventReadTimeout
- The timeout- Returns:
- A builder to configure and create a streaming reader.
-
withMaxOutstandingCheckpointRequest
public B withMaxOutstandingCheckpointRequest(int maxOutstandingCheckpointRequest)
Configures the maximum outstanding checkpoint requests to Pravega (default=3). Upon requesting more checkpoints than the specified maximum, (say a checkpoint request times out on the ReaderCheckpointHook but Pravega is still working on it), this configurations allows Pravega to limit any further checkpoint request being made to the ReaderGroup. This configuration is particularly relevant when multiple checkpoint requests need to be honored (e.g., frequent savepoint requests being triggered concurrently).- Parameters:
maxOutstandingCheckpointRequest
- maximum outstanding checkpoint request.- Returns:
- A builder to configure and create a streaming reader.
-
getDeserializationSchema
protected abstract org.apache.flink.api.common.serialization.DeserializationSchema<T> getDeserializationSchema()
-
getAssignerWithTimeWindows
protected abstract org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>> getAssignerWithTimeWindows()
-
buildSourceFunction
protected FlinkPravegaReader<T> buildSourceFunction()
Builds aFlinkPravegaReader
based on the configuration. Be sure to callinitialize()
before returning the reader to user code.- Returns:
- an uninitiailized reader as a source function.
- Throws:
java.lang.IllegalStateException
- if the configuration is invalid.
-
buildReaderGroupInfo
public AbstractStreamingReaderBuilder.ReaderGroupInfo buildReaderGroupInfo()
Build reader group configuration.
-
generateUid
public java.lang.String generateUid()
Generate a UID for the source, to distinguish the state associated with the checkpoint hook. A good generated UID will: 1. be stable across savepoints for the same inputs 2. disambiguate one source from another (e.g. in a program that uses numerous instances ofFlinkPravegaReader
) 3. allow for reconfiguration of the stream cuts and timeouts- Returns:
- A generated reader group ID.
-
-