Package io.pravega.connectors.flink

Class FlinkPravegaInputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,PravegaInputSplit>

io.pravega.connectors.flink.FlinkPravegaInputFormat<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.api.common.io.InputFormat<T,PravegaInputSplit>, org.apache.flink.core.io.InputSplitSource<PravegaInputSplit>

public class FlinkPravegaInputFormat<T>
extends org.apache.flink.api.common.io.RichInputFormat<T,PravegaInputSplit>

A Flink InputFormat that can be added as a source to read from Pravega in a Flink batch job.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FlinkPravegaInputFormat.Builder<T>	A builder for FlinkPravegaInputFormat to read Pravega streams using the Flink batch API.

Constructor Summary

Constructors

Constructor	Description
FlinkPravegaInputFormat(io.pravega.client.ClientConfig clientConfig, java.util.List<StreamWithBoundaries> streams, org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema)	Creates a new Flink Pravega InputFormat which can be added as a source to a Flink batch job.

Method Summary

Modifier and Type	Method	Description
static <T> FlinkPravegaInputFormat.Builder<T>	builder()	Gets a builder FlinkPravegaInputFormat to read Pravega streams using the Flink batch API.
void	close()
void	closeInputFormat()
void	configure(org.apache.flink.configuration.Configuration parameters)
PravegaInputSplit[]	createInputSplits(int minNumSplits)
protected io.pravega.client.BatchClientFactory	getBatchClientFactory(java.lang.String clientScope, io.pravega.client.ClientConfig clientConfig)
org.apache.flink.core.io.InputSplitAssigner	getInputSplitAssigner(PravegaInputSplit[] inputSplits)
org.apache.flink.api.common.io.statistics.BaseStatistics	getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)
T	nextRecord(T t)
void	open(PravegaInputSplit split)
void	openInputFormat()
boolean	reachedEnd()

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlinkPravegaInputFormat

public FlinkPravegaInputFormat(io.pravega.client.ClientConfig clientConfig,
                               java.util.List<StreamWithBoundaries> streams,
                               org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema)

Creates a new Flink Pravega InputFormat which can be added as a source to a Flink batch job.

Parameters:

clientConfig - The pravega client configuration.

streams - The list of streams to read events from.

deserializationSchema - The implementation to deserialize events from pravega streams.

Method Detail

openInputFormat

public void openInputFormat()
                     throws java.io.IOException

Overrides:

openInputFormat in class org.apache.flink.api.common.io.RichInputFormat<T,PravegaInputSplit>

Throws:

java.io.IOException

getBatchClientFactory

@VisibleForTesting
protected io.pravega.client.BatchClientFactory getBatchClientFactory(java.lang.String clientScope,
                                                                     io.pravega.client.ClientConfig clientConfig)

closeInputFormat

public void closeInputFormat()
                      throws java.io.IOException

Overrides:

closeInputFormat in class org.apache.flink.api.common.io.RichInputFormat<T,PravegaInputSplit>

Throws:

java.io.IOException

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

getStatistics

public org.apache.flink.api.common.io.statistics.BaseStatistics getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)
                                                                       throws java.io.IOException

Throws:

java.io.IOException

createInputSplits

public PravegaInputSplit[] createInputSplits(int minNumSplits)
                                      throws java.io.IOException

Throws:

java.io.IOException

getInputSplitAssigner

public org.apache.flink.core.io.InputSplitAssigner getInputSplitAssigner(PravegaInputSplit[] inputSplits)

open

public void open(PravegaInputSplit split)
          throws java.io.IOException

Throws:

java.io.IOException

reachedEnd

public boolean reachedEnd()
                   throws java.io.IOException

Throws:

java.io.IOException

nextRecord

public T nextRecord(T t)
             throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException

builder

public static <T> FlinkPravegaInputFormat.Builder<T> builder()

Gets a builder FlinkPravegaInputFormat to read Pravega streams using the Flink batch API.

Type Parameters:

T - the element type.

Returns:

A builder to configure and create a batch reader.