org.apache.flink.streaming.api.datastream

类 DataStream<T>

java.lang.Object

org.apache.flink.streaming.api.datastream.DataStream<T>

类型参数:

T - The type of the elements in this stream.

直接已知子类:

KeyedStream, SingleOutputStreamOperator, SplitStream

@Public
public class DataStream<T>
extends Object

A DataStream represents a stream of elements of the same type. A DataStream can be transformed into another DataStream by applying a transformation as for example:

• map(org.apache.flink.api.common.functions.MapFunction<T, R>)
• filter(org.apache.flink.api.common.functions.FilterFunction<T>)

字段概要

字段

限定符和类型	字段和说明
protected StreamExecutionEnvironment	environment
protected org.apache.flink.api.dag.Transformation<T>	transformation

构造器概要

构造器

构造器和说明
DataStream(StreamExecutionEnvironment environment, org.apache.flink.api.dag.Transformation<T> transformation) Create a new DataStream in the given execution environment with partitioning set to forward by default.

方法概要

限定符和类型	方法和说明
DataStreamSink<T>	addSink(SinkFunction<T> sinkFunction) Adds the given sink to this DataStream.
SingleOutputStreamOperator<T>	assignTimestamps(TimestampExtractor<T> extractor) 已过时。 Please use assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks) of assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks) instead.
SingleOutputStreamOperator<T>	assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks<T> timestampAndWatermarkAssigner) Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.
SingleOutputStreamOperator<T>	assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks<T> timestampAndWatermarkAssigner) Assigns timestamps to the elements in the data stream and creates watermarks to signal event time progress based on the elements themselves.
DataStream<T>	broadcast() Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
BroadcastStream<T>	broadcast(org.apache.flink.api.common.state.MapStateDescriptor<?,?>... broadcastStateDescriptors) Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
protected <F> F	clean(F f) Invokes the ClosureCleaner on the given function if closure cleaning is enabled in the ExecutionConfig.
<T2> CoGroupedStreams<T,T2>	coGroup(DataStream<T2> otherStream) Creates a join operation.
<R> BroadcastConnectedStream<T,R>	connect(BroadcastStream<R> broadcastStream) Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.
<R> ConnectedStreams<T,R>	connect(DataStream<R> dataStream) Creates a new ConnectedStreams by connecting DataStream outputs of (possible) different types with each other.
AllWindowedStream<T,GlobalWindow>	countWindowAll(long size) Windows this DataStream into tumbling count windows.
AllWindowedStream<T,GlobalWindow>	countWindowAll(long size, long slide) Windows this DataStream into sliding count windows.
protected <R> SingleOutputStreamOperator<R>	doTransform(String operatorName, org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo, StreamOperatorFactory<R> operatorFactory)
SingleOutputStreamOperator<T>	filter(org.apache.flink.api.common.functions.FilterFunction<T> filter) Applies a Filter transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	flatMap(org.apache.flink.api.common.functions.FlatMapFunction<T,R> flatMapper) Applies a FlatMap transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	flatMap(org.apache.flink.api.common.functions.FlatMapFunction<T,R> flatMapper, org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType) Applies a FlatMap transformation on a DataStream.
DataStream<T>	forward() Sets the partitioning of the DataStream so that the output elements are forwarded to the local subtask of the next operation.
org.apache.flink.api.common.ExecutionConfig	getExecutionConfig()
StreamExecutionEnvironment	getExecutionEnvironment() Returns the StreamExecutionEnvironment that was used to create this DataStream.
int	getId() Returns the ID of the DataStream in the current StreamExecutionEnvironment.
org.apache.flink.api.common.operators.ResourceSpec	getMinResources() Gets the minimum resources for this operator.
int	getParallelism() Gets the parallelism for this operator.
org.apache.flink.api.common.operators.ResourceSpec	getPreferredResources() Gets the preferred resources for this operator.
org.apache.flink.api.dag.Transformation<T>	getTransformation() Returns the Transformation that represents the operation that logically creates this DataStream.
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getType() Gets the type of the stream.
DataStream<T>	global() Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator.
IterativeStream<T>	iterate() Initiates an iterative part of the program that feeds back data streams.
IterativeStream<T>	iterate(long maxWaitTimeMillis) Initiates an iterative part of the program that feeds back data streams.
<T2> JoinedStreams<T,T2>	join(DataStream<T2> otherStream) Creates a join operation.
KeyedStream<T,org.apache.flink.api.java.tuple.Tuple>	keyBy(int... fields) Partitions the operator state of a DataStream by the given key positions.
<K> KeyedStream<T,K>	keyBy(org.apache.flink.api.java.functions.KeySelector<T,K> key) It creates a new KeyedStream that uses the provided key for partitioning its operator states.
<K> KeyedStream<T,K>	keyBy(org.apache.flink.api.java.functions.KeySelector<T,K> key, org.apache.flink.api.common.typeinfo.TypeInformation<K> keyType) It creates a new KeyedStream that uses the provided key with explicit type information for partitioning its operator states.
KeyedStream<T,org.apache.flink.api.java.tuple.Tuple>	keyBy(String... fields) Partitions the operator state of a DataStream using field expressions.
<R> SingleOutputStreamOperator<R>	map(org.apache.flink.api.common.functions.MapFunction<T,R> mapper) Applies a Map transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	map(org.apache.flink.api.common.functions.MapFunction<T,R> mapper, org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType) Applies a Map transformation on a DataStream.
<K> DataStream<T>	partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner, int field) Partitions a tuple DataStream on the specified key fields using a custom partitioner.
<K> DataStream<T>	partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner, org.apache.flink.api.java.functions.KeySelector<T,K> keySelector) Partitions a DataStream on the key returned by the selector, using a custom partitioner.
<K> DataStream<T>	partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner, String field) Partitions a POJO DataStream on the specified key fields using a custom partitioner.
DataStreamSink<T>	print() Writes a DataStream to the standard output stream (stdout).
DataStreamSink<T>	print(String sinkIdentifier) Writes a DataStream to the standard output stream (stdout).
DataStreamSink<T>	printToErr() Writes a DataStream to the standard output stream (stderr).
DataStreamSink<T>	printToErr(String sinkIdentifier) Writes a DataStream to the standard output stream (stderr).
<R> SingleOutputStreamOperator<R>	process(ProcessFunction<T,R> processFunction) Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
<R> SingleOutputStreamOperator<R>	process(ProcessFunction<T,R> processFunction, org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType) Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
<R extends org.apache.flink.api.java.tuple.Tuple> SingleOutputStreamOperator<R>	project(int... fieldIndexes) Initiates a Project transformation on a Tuple DataStream.
DataStream<T>	rebalance() Sets the partitioning of the DataStream so that the output elements are distributed evenly to instances of the next operation in a round-robin fashion.
DataStream<T>	rescale() Sets the partitioning of the DataStream so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion.
protected DataStream<T>	setConnectionType(StreamPartitioner<T> partitioner) Internal function for setting the partitioner for the DataStream.
DataStream<T>	shuffle() Sets the partitioning of the DataStream so that the output elements are shuffled uniformly randomly to the next operation.
SplitStream<T>	split(OutputSelector<T> outputSelector) 已过时。 Please use side output instead.
AllWindowedStream<T,TimeWindow>	timeWindowAll(Time size) Windows this DataStream into tumbling time windows.
AllWindowedStream<T,TimeWindow>	timeWindowAll(Time size, Time slide) Windows this DataStream into sliding time windows.
<R> SingleOutputStreamOperator<R>	transform(String operatorName, org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo, OneInputStreamOperator<T,R> operator) Method for passing user defined operators along with the type information that will transform the DataStream.
<R> SingleOutputStreamOperator<R>	transform(String operatorName, org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo, OneInputStreamOperatorFactory<T,R> operatorFactory) Method for passing user defined operators created by the given factory along with the type information that will transform the DataStream.
DataStream<T>	union(DataStream<T>... streams) Creates a new DataStream by merging DataStream outputs of the same type with each other.
<W extends Window> AllWindowedStream<T,W>	windowAll(WindowAssigner<? super T,W> assigner) Windows this data stream to a AllWindowedStream, which evaluates windows over a non key grouped stream.
DataStreamSink<T>	writeAsCsv(String path) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsCsv(String path, org.apache.flink.core.fs.FileSystem.WriteMode writeMode) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
<X extends org.apache.flink.api.java.tuple.Tuple> DataStreamSink<T>	writeAsCsv(String path, org.apache.flink.core.fs.FileSystem.WriteMode writeMode, String rowDelimiter, String fieldDelimiter) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsText(String path) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsText(String path, org.apache.flink.core.fs.FileSystem.WriteMode writeMode) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeToSocket(String hostName, int port, org.apache.flink.api.common.serialization.SerializationSchema<T> schema) Writes the DataStream to a socket as a byte array.
DataStreamSink<T>	writeUsingOutputFormat(org.apache.flink.api.common.io.OutputFormat<T> format) 已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

environment

protected final StreamExecutionEnvironment environment

transformation

protected final org.apache.flink.api.dag.Transformation<T> transformation

构造器详细资料

DataStream

public DataStream(StreamExecutionEnvironment environment,
                  org.apache.flink.api.dag.Transformation<T> transformation)

Create a new DataStream in the given execution environment with partitioning set to forward by default.

参数:

environment - The StreamExecutionEnvironment

方法详细资料

getId

@Internal
public int getId()

Returns the ID of the DataStream in the current StreamExecutionEnvironment.

返回:

ID of the DataStream

getParallelism

public int getParallelism()

Gets the parallelism for this operator.

返回:

The parallelism set for this operator.

getMinResources

@PublicEvolving
public org.apache.flink.api.common.operators.ResourceSpec getMinResources()

Gets the minimum resources for this operator.

返回:

The minimum resources set for this operator.

getPreferredResources

@PublicEvolving
public org.apache.flink.api.common.operators.ResourceSpec getPreferredResources()

Gets the preferred resources for this operator.

返回:

The preferred resources set for this operator.

getType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getType()

Gets the type of the stream.

返回:

The type of the datastream.

clean

protected <F> F clean(F f)

Invokes the ClosureCleaner on the given function if closure cleaning is enabled in the ExecutionConfig.

返回:

The cleaned Function

getExecutionEnvironment

public StreamExecutionEnvironment getExecutionEnvironment()

Returns the StreamExecutionEnvironment that was used to create this DataStream.

返回:

The Execution Environment

getExecutionConfig

public org.apache.flink.api.common.ExecutionConfig getExecutionConfig()

union

@SafeVarargs
public final DataStream<T> union(DataStream<T>... streams)

Creates a new DataStream by merging DataStream outputs of the same type with each other. The DataStreams merged using this operator will be transformed simultaneously.

参数:

streams - The DataStreams to union output with.

返回:

The DataStream.

split

@Deprecated
public SplitStream<T> split(OutputSelector<T> outputSelector)

已过时。 Please use side output instead.

Operator used for directing tuples to specific named outputs using an OutputSelector. Calling this method on an operator creates a new SplitStream.

参数:

outputSelector - The user defined OutputSelector for directing the tuples.

返回:

The SplitStream

connect

public <R> ConnectedStreams<T,R> connect(DataStream<R> dataStream)

Creates a new ConnectedStreams by connecting DataStream outputs of (possible) different types with each other. The DataStreams connected using this operator can be used with CoFunctions to apply joint transformations.

参数:

dataStream - The DataStream with which this stream will be connected.

返回:

The ConnectedStreams.

connect

@PublicEvolving
public <R> BroadcastConnectedStream<T,R> connect(BroadcastStream<R> broadcastStream)

Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.

The latter can be created using the broadcast(MapStateDescriptor[]) method.

The resulting stream can be further processed using the BroadcastConnectedStream.process(MyFunction) method, where MyFunction can be either a KeyedBroadcastProcessFunction or a BroadcastProcessFunction depending on the current stream being a KeyedStream or not.

参数:

broadcastStream - The broadcast stream with the broadcast state to be connected with this stream.

返回:

The BroadcastConnectedStream.

keyBy

public <K> KeyedStream<T,K> keyBy(org.apache.flink.api.java.functions.KeySelector<T,K> key)

It creates a new KeyedStream that uses the provided key for partitioning its operator states.

参数:

key - The KeySelector to be used for extracting the key for partitioning

返回:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

public <K> KeyedStream<T,K> keyBy(org.apache.flink.api.java.functions.KeySelector<T,K> key,
                                  org.apache.flink.api.common.typeinfo.TypeInformation<K> keyType)

It creates a new KeyedStream that uses the provided key with explicit type information for partitioning its operator states.

参数:

key - The KeySelector to be used for extracting the key for partitioning.

keyType - The type information describing the key type.

返回:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

public KeyedStream<T,org.apache.flink.api.java.tuple.Tuple> keyBy(int... fields)

Partitions the operator state of a DataStream by the given key positions.

参数:

fields - The position of the fields on which the DataStream will be grouped.

返回:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

public KeyedStream<T,org.apache.flink.api.java.tuple.Tuple> keyBy(String... fields)

Partitions the operator state of a DataStream using field expressions. A field expression is either the name of a public field or a getter method with parentheses of the DataStream's underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

参数:

fields - One or more field expressions on which the state of the DataStream operators will be partitioned.

返回:

The DataStream with partitioned state (i.e. KeyedStream)

partitionCustom

public <K> DataStream<T> partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner,
                                         int field)

Partitions a tuple DataStream on the specified key fields using a custom partitioner. This method takes the key position to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys.

参数:

partitioner - The partitioner to assign partitions to keys.

field - The field index on which the DataStream is partitioned.

返回:

The partitioned DataStream.

partitionCustom

public <K> DataStream<T> partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner,
                                         String field)

Partitions a POJO DataStream on the specified key fields using a custom partitioner. This method takes the key expression to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys.

参数:

partitioner - The partitioner to assign partitions to keys.

field - The expression for the field on which the DataStream is partitioned.

返回:

The partitioned DataStream.

partitionCustom

public <K> DataStream<T> partitionCustom(org.apache.flink.api.common.functions.Partitioner<K> partitioner,
                                         org.apache.flink.api.java.functions.KeySelector<T,K> keySelector)

Partitions a DataStream on the key returned by the selector, using a custom partitioner. This method takes the key selector to get the key to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys, i.e. the selector cannot return tuples of fields.

参数:

partitioner - The partitioner to assign partitions to keys.

keySelector - The KeySelector with which the DataStream is partitioned.

返回:

The partitioned DataStream.

另请参阅:

KeySelector

broadcast

public DataStream<T> broadcast()

Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.

返回:

The DataStream with broadcast partitioning set.

broadcast

@PublicEvolving
public BroadcastStream<T> broadcast(org.apache.flink.api.common.state.MapStateDescriptor<?,?>... broadcastStateDescriptors)

Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation. In addition, it implicitly as many broadcast states as the specified descriptors which can be used to store the element of the stream.

参数:

broadcastStateDescriptors - the descriptors of the broadcast states to create.

返回:

A BroadcastStream which can be used in the connect(BroadcastStream) to create a BroadcastConnectedStream for further processing of the elements.

shuffle

@PublicEvolving
public DataStream<T> shuffle()

Sets the partitioning of the DataStream so that the output elements are shuffled uniformly randomly to the next operation.

返回:

The DataStream with shuffle partitioning set.

forward

public DataStream<T> forward()

Sets the partitioning of the DataStream so that the output elements are forwarded to the local subtask of the next operation.

返回:

The DataStream with forward partitioning set.

rebalance

public DataStream<T> rebalance()

Sets the partitioning of the DataStream so that the output elements are distributed evenly to instances of the next operation in a round-robin fashion.

返回:

The DataStream with rebalance partitioning set.

rescale

@PublicEvolving
public DataStream<T> rescale()

Sets the partitioning of the DataStream so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion.

The subset of downstream operations to which the upstream operation sends elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 4, then one upstream operation would distribute elements to two downstream operations while the other upstream operation would distribute to the other two downstream operations. If, on the other hand, the downstream operation has parallelism 2 while the upstream operation has parallelism 4 then two upstream operations will distribute to one downstream operation while the other two upstream operations will distribute to the other downstream operations.

In cases where the different parallelisms are not multiples of each other one or several downstream operations will have a differing number of inputs from upstream operations.

返回:

The DataStream with rescale partitioning set.

global

@PublicEvolving
public DataStream<T> global()

Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator. Use this setting with care since it might cause a serious performance bottleneck in the application.

返回:

The DataStream with shuffle partitioning set.

iterate

@PublicEvolving
public IterativeStream<T> iterate()

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling IterativeStream.closeWith(DataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the IterativeStream.closeWith(DataStream) method is the data stream that will be fed back and used as the input for the iteration head. The user can also use different feedback type than the input of the iteration and treat the input and feedback streams as a ConnectedStreams be calling IterativeStream.withFeedbackType(TypeInformation)

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to split(OutputSelector) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the IterativeStream.closeWith(DataStream) call.

By default a DataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

返回:

The iterative data stream created.

iterate

@PublicEvolving
public IterativeStream<T> iterate(long maxWaitTimeMillis)

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling IterativeStream.closeWith(DataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the IterativeStream.closeWith(DataStream) method is the data stream that will be fed back and used as the input for the iteration head. The user can also use different feedback type than the input of the iteration and treat the input and feedback streams as a ConnectedStreams be calling IterativeStream.withFeedbackType(TypeInformation)

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to split(OutputSelector) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the IterativeStream.closeWith(DataStream) call.

By default a DataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

参数:

maxWaitTimeMillis - Number of milliseconds to wait between inputs before shutting down

返回:

The iterative data stream created.

map

public <R> SingleOutputStreamOperator<R> map(org.apache.flink.api.common.functions.MapFunction<T,R> mapper)

Applies a Map transformation on a DataStream. The transformation calls a MapFunction for each element of the DataStream. Each MapFunction call returns exactly one element. The user can also extend RichMapFunction to gain access to other features provided by the RichFunction interface.

类型参数:

R - output type

参数:

mapper - The MapFunction that is called for each element of the DataStream.

返回:

The transformed DataStream.

map

public <R> SingleOutputStreamOperator<R> map(org.apache.flink.api.common.functions.MapFunction<T,R> mapper,
                                             org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType)

Applies a Map transformation on a DataStream. The transformation calls a MapFunction for each element of the DataStream. Each MapFunction call returns exactly one element. The user can also extend RichMapFunction to gain access to other features provided by the RichFunction interface.

类型参数:

R - output type

参数:

mapper - The MapFunction that is called for each element of the DataStream.

outputType - TypeInformation for the result type of the function.

返回:

The transformed DataStream.

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(org.apache.flink.api.common.functions.FlatMapFunction<T,R> flatMapper)

Applies a FlatMap transformation on a DataStream. The transformation calls a FlatMapFunction for each element of the DataStream. Each FlatMapFunction call can return any number of elements including none. The user can also extend RichFlatMapFunction to gain access to other features provided by the RichFunction interface.

类型参数:

R - output type

参数:

flatMapper - The FlatMapFunction that is called for each element of the DataStream

返回:

The transformed DataStream.

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(org.apache.flink.api.common.functions.FlatMapFunction<T,R> flatMapper,
                                                 org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType)

Applies a FlatMap transformation on a DataStream. The transformation calls a FlatMapFunction for each element of the DataStream. Each FlatMapFunction call can return any number of elements including none. The user can also extend RichFlatMapFunction to gain access to other features provided by the RichFunction interface.

类型参数:

R - output type

参数:

flatMapper - The FlatMapFunction that is called for each element of the DataStream

outputType - TypeInformation for the result type of the function.

返回:

The transformed DataStream.

process

@PublicEvolving
public <R> SingleOutputStreamOperator<R> process(ProcessFunction<T,R> processFunction)

Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements.

类型参数:

R - The type of elements emitted by the ProcessFunction.

参数:

processFunction - The ProcessFunction that is called for each element in the stream.

返回:

The transformed DataStream.

process

@Internal
public <R> SingleOutputStreamOperator<R> process(ProcessFunction<T,R> processFunction,
                                                           org.apache.flink.api.common.typeinfo.TypeInformation<R> outputType)

Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements.

类型参数:

R - The type of elements emitted by the ProcessFunction.

参数:

processFunction - The ProcessFunction that is called for each element in the stream.

outputType - TypeInformation for the result type of the function.

返回:

The transformed DataStream.

filter

public SingleOutputStreamOperator<T> filter(org.apache.flink.api.common.functions.FilterFunction<T> filter)

Applies a Filter transformation on a DataStream. The transformation calls a FilterFunction for each element of the DataStream and retains only those element for which the function returns true. Elements for which the function returns false are filtered. The user can also extend RichFilterFunction to gain access to other features provided by the RichFunction interface.

参数:

filter - The FilterFunction that is called for each element of the DataStream.

返回:

The filtered DataStream.

project

@PublicEvolving
public <R extends org.apache.flink.api.java.tuple.Tuple> SingleOutputStreamOperator<R> project(int... fieldIndexes)

Initiates a Project transformation on a Tuple DataStream.
Note: Only Tuple DataStreams can be projected.

The transformation projects each Tuple of the DataSet onto a (sub)set of fields.

参数:

fieldIndexes - The field indexes of the input tuples that are retained. The order of fields in the output tuple corresponds to the order of field indexes.

返回:

The projected DataStream

另请参阅:

Tuple, DataStream

coGroup

public <T2> CoGroupedStreams<T,T2> coGroup(DataStream<T2> otherStream)

Creates a join operation. See CoGroupedStreams for an example of how the keys and window can be specified.

join

public <T2> JoinedStreams<T,T2> join(DataStream<T2> otherStream)

Creates a join operation. See JoinedStreams for an example of how the keys and window can be specified.

timeWindowAll

public AllWindowedStream<T,TimeWindow> timeWindowAll(Time size)

Windows this DataStream into tumbling time windows.

This is a shortcut for either .window(TumblingEventTimeWindows.of(size)) or .window(TumblingProcessingTimeWindows.of(size)) depending on the time characteristic set using

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance. StreamExecutionEnvironment.setStreamTimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic)

参数:

size - The size of the window.

timeWindowAll

public AllWindowedStream<T,TimeWindow> timeWindowAll(Time size,
                                                     Time slide)

Windows this DataStream into sliding time windows.

This is a shortcut for either .window(SlidingEventTimeWindows.of(size, slide)) or .window(SlidingProcessingTimeWindows.of(size, slide)) depending on the time characteristic set using StreamExecutionEnvironment.setStreamTimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic)

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

参数:

size - The size of the window.

countWindowAll

public AllWindowedStream<T,GlobalWindow> countWindowAll(long size)

Windows this DataStream into tumbling count windows.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

参数:

size - The size of the windows in number of elements.

countWindowAll

public AllWindowedStream<T,GlobalWindow> countWindowAll(long size,
                                                        long slide)

Windows this DataStream into sliding count windows.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

参数:

size - The size of the windows in number of elements.

slide - The slide interval in number of elements.

windowAll

@PublicEvolving
public <W extends Window> AllWindowedStream<T,W> windowAll(WindowAssigner<? super T,W> assigner)

Windows this data stream to a AllWindowedStream, which evaluates windows over a non key grouped stream. Elements are put into windows by a WindowAssigner. The grouping of elements is done by window.

A Trigger can be defined to specify when windows are evaluated. However, WindowAssigners have a default Trigger that is used if a Trigger is not specified.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

参数:

assigner - The WindowAssigner that assigns elements to windows.

返回:

The trigger windows data stream.

assignTimestamps

@Deprecated
public SingleOutputStreamOperator<T> assignTimestamps(TimestampExtractor<T> extractor)

已过时。 Please use assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks) of assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks) instead.

Extracts a timestamp from an element and assigns it as the internal timestamp of that element. The internal timestamps are, for example, used to to event-time window operations.

If you know that the timestamps are strictly increasing you can use an AscendingTimestampExtractor. Otherwise, you should provide a TimestampExtractor that also implements TimestampExtractor.getCurrentWatermark() to keep track of watermarks.

参数:

extractor - The TimestampExtractor that is called for each element of the DataStream.

另请参阅:

assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks), assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)

assignTimestampsAndWatermarks

public SingleOutputStreamOperator<T> assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks<T> timestampAndWatermarkAssigner)

Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.

This method creates watermarks periodically (for example every second), based on the watermarks indicated by the given watermark generator. Even when no new elements in the stream arrive, the given watermark generator will be periodically checked for new watermarks. The interval in which watermarks are generated is defined in ExecutionConfig.setAutoWatermarkInterval(long).

Use this method for the common cases, where some characteristic over all elements should generate the watermarks, or where watermarks are simply trailing behind the wall clock time by a certain amount.

For the second case and when the watermarks are required to lag behind the maximum timestamp seen so far in the elements of the stream by a fixed amount of time, and this amount is known in advance, use the BoundedOutOfOrdernessTimestampExtractor.

For cases where watermarks should be created in an irregular fashion, for example based on certain markers that some element carry, use the AssignerWithPunctuatedWatermarks.

参数:

timestampAndWatermarkAssigner - The implementation of the timestamp assigner and watermark generator.

返回:

The stream after the transformation, with assigned timestamps and watermarks.

另请参阅:

AssignerWithPeriodicWatermarks, AssignerWithPunctuatedWatermarks, assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks)

assignTimestampsAndWatermarks

public SingleOutputStreamOperator<T> assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks<T> timestampAndWatermarkAssigner)

Assigns timestamps to the elements in the data stream and creates watermarks to signal event time progress based on the elements themselves.

This method creates watermarks based purely on stream elements. For each element that is handled via TimestampAssigner.extractTimestamp(Object, long), the AssignerWithPunctuatedWatermarks.checkAndGetNextWatermark(Object, long) method is called, and a new watermark is emitted, if the returned watermark value is non-negative and greater than the previous watermark.

This method is useful when the data stream embeds watermark elements, or certain elements carry a marker that can be used to determine the current event time watermark. This operation gives the programmer full control over the watermark generation. Users should be aware that too aggressive watermark generation (i.e., generating hundreds of watermarks every second) can cost some performance.

For cases where watermarks should be created in a regular fashion, for example every x milliseconds, use the AssignerWithPeriodicWatermarks.

参数:

timestampAndWatermarkAssigner - The implementation of the timestamp assigner and watermark generator.

返回:

The stream after the transformation, with assigned timestamps and watermarks.

另请参阅:

AssignerWithPunctuatedWatermarks, AssignerWithPeriodicWatermarks, assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks)

print

@PublicEvolving
public DataStreamSink<T> print()

Writes a DataStream to the standard output stream (stdout).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stdout on the machine where the code is executed, i.e. the Flink worker.

返回:

The closed DataStream.

printToErr

@PublicEvolving
public DataStreamSink<T> printToErr()

Writes a DataStream to the standard output stream (stderr).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stderr on the machine where the code is executed, i.e. the Flink worker.

返回:

The closed DataStream.

print

@PublicEvolving
public DataStreamSink<T> print(String sinkIdentifier)

Writes a DataStream to the standard output stream (stdout).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stdout on the machine where the code is executed, i.e. the Flink worker.

参数:

sinkIdentifier - The string to prefix the output with.

返回:

The closed DataStream.

printToErr

@PublicEvolving
public DataStreamSink<T> printToErr(String sinkIdentifier)

Writes a DataStream to the standard output stream (stderr).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stderr on the machine where the code is executed, i.e. the Flink worker.

参数:

sinkIdentifier - The string to prefix the output with.

返回:

The closed DataStream.

writeAsText

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsText(String path)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by path in text format.

For every element of the DataStream the result of Object.toString() is written.

参数:

path - The path pointing to the location the text file is written to.

返回:

The closed DataStream.

writeAsText

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsText(String path,
                                                                  org.apache.flink.core.fs.FileSystem.WriteMode writeMode)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by path in text format.

For every element of the DataStream the result of Object.toString() is written.

参数:

path - The path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

返回:

The closed DataStream.

writeAsCsv

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsCsv(String path)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

参数:

path - the path pointing to the location the text file is written to

返回:

the closed DataStream

writeAsCsv

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsCsv(String path,
                                                                 org.apache.flink.core.fs.FileSystem.WriteMode writeMode)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

参数:

path - the path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

返回:

the closed DataStream

writeAsCsv

@Deprecated
 @PublicEvolving
public <X extends org.apache.flink.api.java.tuple.Tuple> DataStreamSink<T> writeAsCsv(String path,
                                                                                                                   org.apache.flink.core.fs.FileSystem.WriteMode writeMode,
                                                                                                                   String rowDelimiter,
                                                                                                                   String fieldDelimiter)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter. The writing is performed periodically every millis milliseconds.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

参数:

path - the path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

rowDelimiter - the delimiter for two rows

fieldDelimiter - the delimiter for two fields

返回:

the closed DataStream

writeToSocket

@PublicEvolving
public DataStreamSink<T> writeToSocket(String hostName,
                                                       int port,
                                                       org.apache.flink.api.common.serialization.SerializationSchema<T> schema)

Writes the DataStream to a socket as a byte array. The format of the output is specified by a SerializationSchema.

参数:

hostName - host of the socket

port - port of the socket

schema - schema for serialization

返回:

the closed DataStream

writeUsingOutputFormat

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeUsingOutputFormat(org.apache.flink.api.common.io.OutputFormat<T> format)

已过时。 Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes the dataStream into an output, described by an OutputFormat.

The output is not participating in Flink's checkpointing!

For writing to a file system periodically, the use of the "flink-connector-filesystem" is recommended.

参数:

format - The output format

返回:

The closed DataStream

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String operatorName,
                                                                   org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo,
                                                                   OneInputStreamOperator<T,R> operator)

Method for passing user defined operators along with the type information that will transform the DataStream.

类型参数:

R - type of the return stream

参数:

operatorName - name of the operator, for logging purposes

outTypeInfo - the output type of the operator

operator - the object containing the transformation logic

返回:

the data stream constructed

另请参阅:

transform(String, TypeInformation, OneInputStreamOperatorFactory)

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String operatorName,
                                                                   org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo,
                                                                   OneInputStreamOperatorFactory<T,R> operatorFactory)

Method for passing user defined operators created by the given factory along with the type information that will transform the DataStream.

This method uses the rather new operator factories and should only be used when custom factories are needed.

类型参数:

R - type of the return stream

参数:

operatorName - name of the operator, for logging purposes

outTypeInfo - the output type of the operator

operatorFactory - the factory for the operator.

返回:

the data stream constructed.

doTransform

protected <R> SingleOutputStreamOperator<R> doTransform(String operatorName,
                                                        org.apache.flink.api.common.typeinfo.TypeInformation<R> outTypeInfo,
                                                        StreamOperatorFactory<R> operatorFactory)

setConnectionType

protected DataStream<T> setConnectionType(StreamPartitioner<T> partitioner)

Internal function for setting the partitioner for the DataStream.

参数:

partitioner - Partitioner to set.

返回:

The modified DataStream.

addSink

public DataStreamSink<T> addSink(SinkFunction<T> sinkFunction)

Adds the given sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

参数:

sinkFunction - The object containing the sink's invoke function.

返回:

The closed DataStream.

getTransformation

@Internal
public org.apache.flink.api.dag.Transformation<T> getTransformation()

Returns the Transformation that represents the operation that logically creates this DataStream.

返回:

The Transformation