org.apache.flink.api.java.sampling

类 ReservoirSamplerWithoutReplacement<T>

java.lang.Object

org.apache.flink.api.java.sampling.RandomSampler<T>

org.apache.flink.api.java.sampling.DistributedRandomSampler<T>

org.apache.flink.api.java.sampling.ReservoirSamplerWithoutReplacement<T>

类型参数:

T - The type of the sampler.

@Internal
public class ReservoirSamplerWithoutReplacement<T>
extends DistributedRandomSampler<T>

A simple in memory implementation of Reservoir Sampling without replacement, and with only one pass through the input iteration whose size is unpredictable. The basic idea behind this sampler implementation is to generate a random number for each input element as its weight, select the top K elements with max weight. As the weights are generated randomly, so are the selected top K elements. The algorithm is implemented using the DistributedRandomSampler interface. In the first phase, we generate random numbers as the weights for each element and select top K elements as the output of each partitions. In the second phase, we select top K elements from all the outputs of the first phase.

This implementation refers to the algorithm described in "Optimal Random Sampling from Distributed Streams Revisited".

字段概要

从类继承的字段 org.apache.flink.api.java.sampling.DistributedRandomSampler

emptyIntermediateIterable, numSamples

从类继承的字段 org.apache.flink.api.java.sampling.RandomSampler

emptyIterable, EPSILON

构造器概要

构造器

构造器和说明
ReservoirSamplerWithoutReplacement(int numSamples) Create a new sampler with reservoir size and a default random number generator.
ReservoirSamplerWithoutReplacement(int numSamples, long seed) Create a new sampler with reservoir size and the seed for random number generator.
ReservoirSamplerWithoutReplacement(int numSamples, Random random) Create a new sampler with reservoir size and a supplied random number generator.

方法概要

限定符和类型	方法和说明
Iterator<IntermediateSampleData<T>>	sampleInPartition(Iterator<T> input) Sample algorithm for the first phase.

从类继承的方法 org.apache.flink.api.java.sampling.DistributedRandomSampler

sample, sampleInCoordinator

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

ReservoirSamplerWithoutReplacement

public ReservoirSamplerWithoutReplacement(int numSamples,
                                          Random random)

Create a new sampler with reservoir size and a supplied random number generator.

参数:

numSamples - Maximum number of samples to retain in reservoir, must be non-negative.

random - Instance of random number generator for sampling.

ReservoirSamplerWithoutReplacement

public ReservoirSamplerWithoutReplacement(int numSamples)

Create a new sampler with reservoir size and a default random number generator.

参数:

numSamples - Maximum number of samples to retain in reservoir, must be non-negative.

ReservoirSamplerWithoutReplacement

public ReservoirSamplerWithoutReplacement(int numSamples,
                                          long seed)

Create a new sampler with reservoir size and the seed for random number generator.

参数:

numSamples - Maximum number of samples to retain in reservoir, must be non-negative.

seed - Random number generator seed.

方法详细资料

sampleInPartition

public Iterator<IntermediateSampleData<T>> sampleInPartition(Iterator<T> input)

从类复制的说明: DistributedRandomSampler

Sample algorithm for the first phase. It operates on a single partition.

指定者:

sampleInPartition 在类中 DistributedRandomSampler<T>

参数:

input - The DataSet input of each partition.

返回:

Intermediate sample output which will be used as the input of the second phase.