org.apache.flink.python.chain

Class PythonOperatorChainingOptimizer

java.lang.Object

org.apache.flink.python.chain.PythonOperatorChainingOptimizer

public class PythonOperatorChainingOptimizer
extends Object

A util class which attempts to chain all available Python functions.

An operator could be chained to it's predecessor if all of the following conditions are met:

• Both of them are Python operators
• The parallelism, the maximum parallelism and the slot sharing group are all the same
• The chaining strategy is ChainingStrategy.ALWAYS and the chaining strategy of the predecessor isn't ChainingStrategy.NEVER
• This partitioner between them is ForwardPartitioner

The properties of the generated chained operator are as following:

• The name is the concatenation of all the names of the chained operators
• The parallelism, the maximum parallelism and the slot sharing group are from one of the chained operators as all of them are the same between the chained operators
• The chaining strategy is the same as the head operator
• The uid and the uidHash are the same as the head operator

Constructor Summary

Constructors

Constructor and Description
PythonOperatorChainingOptimizer()

Method Summary

Modifier and Type	Method and Description
static void	apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env) Perform chaining optimization.
static org.apache.flink.api.dag.Transformation<?>	apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, org.apache.flink.api.dag.Transformation<?> transformation) Perform chaining optimization.
static List<org.apache.flink.api.dag.Transformation<?>>	optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations) Perform chaining optimization.
static org.apache.flink.api.java.tuple.Tuple2<List<org.apache.flink.api.dag.Transformation<?>>,org.apache.flink.api.dag.Transformation<?>>	optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations, org.apache.flink.api.dag.Transformation<?> targetTransformation) Perform chaining optimization.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PythonOperatorChainingOptimizer

public PythonOperatorChainingOptimizer()

Method Detail

apply

public static void apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env)
                  throws Exception

Perform chaining optimization. It will iterate the transformations defined in the given StreamExecutionEnvironment and update them with the chained transformations.

Throws:

Exception

apply

public static org.apache.flink.api.dag.Transformation<?> apply(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
                                                               org.apache.flink.api.dag.Transformation<?> transformation)
                                                        throws Exception

Perform chaining optimization. It will iterate the transformations defined in the given StreamExecutionEnvironment and update them with the chained transformations. Besides, it will return the transformation after chaining optimization for the given transformation.

Throws:

Exception

optimize

public static List<org.apache.flink.api.dag.Transformation<?>> optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations)

Perform chaining optimization. It will return the chained transformations for the given transformation list.

optimize

public static org.apache.flink.api.java.tuple.Tuple2<List<org.apache.flink.api.dag.Transformation<?>>,org.apache.flink.api.dag.Transformation<?>> optimize(List<org.apache.flink.api.dag.Transformation<?>> transformations,
                                                                                                                                                           org.apache.flink.api.dag.Transformation<?> targetTransformation)

Perform chaining optimization. It will returns the chained transformations and the transformation after chaining optimization for the given transformation.