org.apache.flink.api.java.sampling

Class DistributedRandomSampler<T>

java.lang.Object

org.apache.flink.api.java.sampling.RandomSampler<T>

org.apache.flink.api.java.sampling.DistributedRandomSampler<T>

Type Parameters:

T - The input data type.

Direct Known Subclasses:

ReservoirSamplerWithoutReplacement, ReservoirSamplerWithReplacement

public abstract class DistributedRandomSampler<T>
extends RandomSampler<T>

For sampling with fraction, the sample algorithms are natively distributed, while it's not true for fixed size sample algorithms. The fixed size sample algorithms require two-phases sampling (according to our current implementation). In the first phase, each distributed partition is sampled independently. The partial sampling results are handled by a central coordinator. The central coordinator combines the partial sampling results to form the final result.

Field Summary

Fields

Modifier and Type	Field and Description
protected Iterator<IntermediateSampleData<T>>	EMPTY_INTERMEDIATE_ITERABLE
protected int	numSamples

Fields inherited from class org.apache.flink.api.java.sampling.RandomSampler

EMPTY_ITERABLE, EPSILON

Constructor Summary

Constructors

Constructor and Description
DistributedRandomSampler(int numSamples)

Method Summary

Modifier and Type	Method and Description
Iterator<T>	sample(Iterator<T> input) Combine the first phase and second phase in sequence, implemented for test purpose only.
Iterator<T>	sampleInCoordinator(Iterator<IntermediateSampleData<T>> input) Sample algorithm for the second phase.
abstract Iterator<IntermediateSampleData<T>>	sampleInPartition(Iterator<T> input) Sample algorithm for the first phase.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

numSamples

protected final int numSamples

EMPTY_INTERMEDIATE_ITERABLE

protected final Iterator<IntermediateSampleData<T>> EMPTY_INTERMEDIATE_ITERABLE

Constructor Detail

DistributedRandomSampler

public DistributedRandomSampler(int numSamples)

Method Detail

sampleInPartition

public abstract Iterator<IntermediateSampleData<T>> sampleInPartition(Iterator<T> input)

Sample algorithm for the first phase. It operates on a single partition.

Parameters:

input - The DataSet input of each partition.

Returns:

Intermediate sample output which will be used as the input of the second phase.

sampleInCoordinator

public Iterator<T> sampleInCoordinator(Iterator<IntermediateSampleData<T>> input)

Sample algorithm for the second phase. This operation should be executed as the UDF of an all reduce operation.

Parameters:

input - The intermediate sample output generated in the first phase.

Returns:

The sampled output.

sample

public Iterator<T> sample(Iterator<T> input)

Combine the first phase and second phase in sequence, implemented for test purpose only.

Specified by:

sample in class RandomSampler<T>

Parameters:

input - Source data.

Returns:

Sample result in sequence.