org.apache.flink.api.java.utils

Class DataSetUtils

java.lang.Object

org.apache.flink.api.java.utils.DataSetUtils

public final class DataSetUtils
extends Object

This class provides simple utility methods for zipping elements in a data set with an index or with a unique identifier.

Method Summary

Methods

Modifier and Type	Method and Description
static int	getBitSize(long value)
static <T> MapPartitionOperator<T,T>	sample(DataSet<T> input, boolean withReplacement, double fraction) Generate a sample of DataSet by the probability fraction of each element.
static <T> MapPartitionOperator<T,T>	sample(DataSet<T> input, boolean withReplacement, double fraction, long seed) Generate a sample of DataSet by the probability fraction of each element.
static <T> DataSet<T>	sampleWithSize(DataSet<T> input, boolean withReplacement, int numSample) Generate a sample of DataSet which contains fixed size elements.
static <T> DataSet<T>	sampleWithSize(DataSet<T> input, boolean withReplacement, int numSample, long seed) Generate a sample of DataSet which contains fixed size elements.
static <T> DataSet<Tuple2<Long,T>>	zipWithIndex(DataSet<T> input) Method that assigns a unique Long value to all elements in the input data set.
static <T> DataSet<Tuple2<Long,T>>	zipWithUniqueId(DataSet<T> input) Method that assigns a unique Long value to all elements in the input data set in the following way: a map function is applied to the input data set each map task holds a counter c which is increased for each record c is shifted by n bits where n = log2(number of parallel tasks) to create a unique ID among all tasks, the task id is added to the counter for each record, the resulting counter is collected

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

zipWithIndex

public static <T> DataSet<Tuple2<Long,T>> zipWithIndex(DataSet<T> input)

Method that assigns a unique Long value to all elements in the input data set. The generated values are consecutive.

Parameters:

input - the input data set

Returns:

a data set of tuple 2 consisting of consecutive ids and initial values.

zipWithUniqueId

public static <T> DataSet<Tuple2<Long,T>> zipWithUniqueId(DataSet<T> input)

Method that assigns a unique Long value to all elements in the input data set in the following way:

• a map function is applied to the input data set
• each map task holds a counter c which is increased for each record
• c is shifted by n bits where n = log2(number of parallel tasks)
• to create a unique ID among all tasks, the task id is added to the counter
• for each record, the resulting counter is collected

Parameters:

input - the input data set

Returns:

a data set of tuple 2 consisting of ids and initial values.

sample

public static <T> MapPartitionOperator<T,T> sample(DataSet<T> input,
                                   boolean withReplacement,
                                   double fraction)

Generate a sample of DataSet by the probability fraction of each element.

Parameters:

withReplacement - Whether element can be selected more than once.

fraction - Probability that each element is chosen, should be [0,1] without replacement, and [0, ∞) with replacement. While fraction is larger than 1, the elements are expected to be selected multi times into sample on average.

Returns:

The sampled DataSet

sample

public static <T> MapPartitionOperator<T,T> sample(DataSet<T> input,
                                   boolean withReplacement,
                                   double fraction,
                                   long seed)

Generate a sample of DataSet by the probability fraction of each element.

Parameters:

withReplacement - Whether element can be selected more than once.

fraction - Probability that each element is chosen, should be [0,1] without replacement, and [0, ∞) with replacement. While fraction is larger than 1, the elements are expected to be selected multi times into sample on average.

seed - random number generator seed.

Returns:

The sampled DataSet

sampleWithSize

public static <T> DataSet<T> sampleWithSize(DataSet<T> input,
                            boolean withReplacement,
                            int numSample)

Generate a sample of DataSet which contains fixed size elements.

NOTE: Sample with fixed size is not as efficient as sample with fraction, use sample with fraction unless you need exact precision.

Parameters:

withReplacement - Whether element can be selected more than once.

numSample - The expected sample size.

Returns:

The sampled DataSet

sampleWithSize

public static <T> DataSet<T> sampleWithSize(DataSet<T> input,
                            boolean withReplacement,
                            int numSample,
                            long seed)

Generate a sample of DataSet which contains fixed size elements.

NOTE: Sample with fixed size is not as efficient as sample with fraction, use sample with fraction unless you need exact precision.

Parameters:

withReplacement - Whether element can be selected more than once.

numSample - The expected sample size.

seed - Random number generator seed.

Returns:

The sampled DataSet

getBitSize

public static int getBitSize(long value)