org.apache.flink.api.java.operators

Class DeltaIteration<ST,WT>

java.lang.Object

org.apache.flink.api.java.operators.DeltaIteration<ST,WT>

Type Parameters:

ST - The data type of the solution set.

WT - The data type of the workset (the feedback data set).

@Public
public class DeltaIteration<ST,WT>
extends Object

The DeltaIteration represents the start of a delta iteration. It is created from the DataSet that represents the initial solution set via the DataSet.iterateDelta(DataSet, int, int...) method.

See Also:

DataSet.iterateDelta(DataSet, int, int...), DataSet.iterateDelta(DataSet, int, int[])

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DeltaIteration.SolutionSetPlaceHolder<ST> A DataSet that acts as a placeholder for the solution set during the iteration.
static class	DeltaIteration.WorksetPlaceHolder<WT> A DataSet that acts as a placeholder for the workset during the iteration.

Constructor Summary

Constructors

Constructor and Description
DeltaIteration(ExecutionEnvironment context, org.apache.flink.api.common.typeinfo.TypeInformation<ST> type, DataSet<ST> solutionSet, DataSet<WT> workset, org.apache.flink.api.common.operators.Keys<ST> keys, int maxIterations)

Method Summary

Modifier and Type	Method and Description
DataSet<ST>	closeWith(DataSet<ST> solutionSetDelta, DataSet<WT> newWorkset) Closes the delta iteration.
org.apache.flink.api.common.aggregators.AggregatorRegistry	getAggregators() Gets the registry for aggregators for the iteration.
DataSet<ST>	getInitialSolutionSet() Gets the initial solution set.
DataSet<WT>	getInitialWorkset() Gets the initial workset.
org.apache.flink.api.common.operators.ResourceSpec	getMinResources() Gets the minimum resources from this iteration.
String	getName() Gets the name from this iteration.
int	getParallelism() Gets the iteration's parallelism.
org.apache.flink.api.common.operators.ResourceSpec	getPreferredResources() Gets the preferred resources from this iteration.
DeltaIteration.SolutionSetPlaceHolder<ST>	getSolutionSet() Gets the solution set of the delta iteration.
DeltaIteration.WorksetPlaceHolder<WT>	getWorkset() Gets the working set of the delta iteration.
boolean	isSolutionSetUnManaged() gets whether the solution set is in managed or unmanaged memory.
DeltaIteration<ST,WT>	name(String name) Sets the name for the iteration.
DeltaIteration<ST,WT>	parallelism(int parallelism) Sets the parallelism for the iteration.
<X extends org.apache.flink.types.Value> DeltaIteration<ST,WT>	registerAggregationConvergenceCriterion(String name, org.apache.flink.api.common.aggregators.Aggregator<X> aggregator, org.apache.flink.api.common.aggregators.ConvergenceCriterion<X> convergenceCheck) Registers an Aggregator for the iteration together with a ConvergenceCriterion.
DeltaIteration<ST,WT>	registerAggregator(String name, org.apache.flink.api.common.aggregators.Aggregator<?> aggregator) Registers an Aggregator for the iteration.
void	setSolutionSetUnManaged(boolean solutionSetUnManaged) Sets whether to keep the solution set in managed memory (safe against heap exhaustion) or unmanaged memory (objects on heap).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeltaIteration

public DeltaIteration(ExecutionEnvironment context,
                      org.apache.flink.api.common.typeinfo.TypeInformation<ST> type,
                      DataSet<ST> solutionSet,
                      DataSet<WT> workset,
                      org.apache.flink.api.common.operators.Keys<ST> keys,
                      int maxIterations)

Method Detail

closeWith

public DataSet<ST> closeWith(DataSet<ST> solutionSetDelta,
                             DataSet<WT> newWorkset)

Closes the delta iteration. This method defines the end of the delta iteration's function.

Parameters:

solutionSetDelta - The delta for the solution set. The delta will be merged into the solution set at the end of each iteration.

newWorkset - The new workset (feedback data set) that will be fed back to the next iteration.

Returns:

The DataSet that represents the result of the iteration, after the computation has terminated.

See Also:

DataSet.iterateDelta(DataSet, int, int...)

getInitialSolutionSet

public DataSet<ST> getInitialSolutionSet()

Gets the initial solution set. This is the data set on which the delta iteration was started.

Consider the following example:

 
 DataSet<MyType> solutionSetData = ...;
 DataSet<AnotherType> worksetData = ...;

 DeltaIteration<MyType, AnotherType> iteration = solutionSetData.iteratorDelta(worksetData, 10, ...);
 
 

The solutionSetData would be the data set returned by iteration.getInitialSolutionSet();.

Returns:

The data set that forms the initial solution set.

getInitialWorkset

public DataSet<WT> getInitialWorkset()

Gets the initial workset. This is the data set passed to the method that starts the delta iteration.

Consider the following example:

 
 DataSet<MyType> solutionSetData = ...;
 DataSet<AnotherType> worksetData = ...;

 DeltaIteration<MyType, AnotherType> iteration = solutionSetData.iteratorDelta(worksetData, 10, ...);
 
 

The worksetData would be the data set returned by iteration.getInitialWorkset();.

Returns:

The data set that forms the initial workset.

getSolutionSet

public DeltaIteration.SolutionSetPlaceHolder<ST> getSolutionSet()

Gets the solution set of the delta iteration. The solution set represents the state that is kept across iterations.

Returns:

The solution set of the delta iteration.

getWorkset

public DeltaIteration.WorksetPlaceHolder<WT> getWorkset()

Gets the working set of the delta iteration. The working set is constructed by the previous iteration.

Returns:

The working set of the delta iteration.

name

public DeltaIteration<ST,WT> name(String name)

Sets the name for the iteration. The name is displayed in logs and messages.

Parameters:

name - The name for the iteration.

Returns:

The iteration object, for function call chaining.

getName

public String getName()

Gets the name from this iteration.

Returns:

The name of the iteration.

parallelism

public DeltaIteration<ST,WT> parallelism(int parallelism)

Sets the parallelism for the iteration.

Parameters:

parallelism - The parallelism.

Returns:

The iteration object, for function call chaining.

getParallelism

public int getParallelism()

Gets the iteration's parallelism.

Returns:

The iteration's parallelism, or ExecutionConfig.PARALLELISM_DEFAULT if not set.

getMinResources

@PublicEvolving
public org.apache.flink.api.common.operators.ResourceSpec getMinResources()

Gets the minimum resources from this iteration. If no minimum resources have been set, it returns the default empty resource.

Returns:

The minimum resources of the iteration.

getPreferredResources

@PublicEvolving
public org.apache.flink.api.common.operators.ResourceSpec getPreferredResources()

Gets the preferred resources from this iteration. If no preferred resources have been set, it returns the default empty resource.

Returns:

The preferred resources of the iteration.

registerAggregator

@PublicEvolving
public DeltaIteration<ST,WT> registerAggregator(String name,
                                                                org.apache.flink.api.common.aggregators.Aggregator<?> aggregator)

Registers an Aggregator for the iteration. Aggregators can be used to maintain simple statistics during the iteration, such as number of elements processed. The aggregators compute global aggregates: After each iteration step, the values are globally aggregated to produce one aggregate that represents statistics across all parallel instances. The value of an aggregator can be accessed in the next iteration.

Aggregators can be accessed inside a function via the AbstractRichFunction.getIterationRuntimeContext() method.

Parameters:

name - The name under which the aggregator is registered.

aggregator - The aggregator class.

Returns:

The DeltaIteration itself, to allow chaining function calls.

registerAggregationConvergenceCriterion

@PublicEvolving
public <X extends org.apache.flink.types.Value> DeltaIteration<ST,WT> registerAggregationConvergenceCriterion(String name,
                                                                                                                              org.apache.flink.api.common.aggregators.Aggregator<X> aggregator,
                                                                                                                              org.apache.flink.api.common.aggregators.ConvergenceCriterion<X> convergenceCheck)

Registers an Aggregator for the iteration together with a ConvergenceCriterion. For a general description of aggregators, see registerAggregator(String, Aggregator) and Aggregator. At the end of each iteration, the convergence criterion takes the aggregator's global aggregate value and decides whether the iteration should terminate. A typical use case is to have an aggregator that sums up the total error of change in an iteration step and have to have a convergence criterion that signals termination as soon as the aggregate value is below a certain threshold.

Parameters:

name - The name under which the aggregator is registered.

aggregator - The aggregator class.

convergenceCheck - The convergence criterion.

Returns:

The DeltaIteration itself, to allow chaining function calls.

getAggregators

@PublicEvolving
public org.apache.flink.api.common.aggregators.AggregatorRegistry getAggregators()

Gets the registry for aggregators for the iteration.

Returns:

The registry with all aggregators.

setSolutionSetUnManaged

public void setSolutionSetUnManaged(boolean solutionSetUnManaged)

Sets whether to keep the solution set in managed memory (safe against heap exhaustion) or unmanaged memory (objects on heap).

Parameters:

solutionSetUnManaged - True to keep the solution set in unmanaged memory, false to keep it in managed memory.

See Also:

isSolutionSetUnManaged()

isSolutionSetUnManaged

public boolean isSolutionSetUnManaged()

gets whether the solution set is in managed or unmanaged memory.

Returns:

True, if the solution set is in unmanaged memory (object heap), false if in managed memory.

See Also:

setSolutionSetUnManaged(boolean)