org.apache.cassandra.utils.streamhist

Class StreamingTombstoneHistogramBuilder

java.lang.Object

org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder

public class StreamingTombstoneHistogramBuilder
extends java.lang.Object

Histogram that can be constructed from streaming of data.

The original algorithm is taken from following paper: Yael Ben-Haim and Elad Tom-Tov, "A Streaming Parallel Decision Tree Algorithm" (2010) http://jmlr.csail.mit.edu/papers/volume11/ben-haim10a/ben-haim10a.pdf

Algorithm: Histogram is represented as collection of {point, weight} pairs. When new point p with weight m is added:

1. If point p is already exists in collection, add m to recorded value of point p
2. If there is no point p in the collection, add point p with weight m
3. If point was added and collection size became lorger than maxBinSize:
1. Find nearest points p1 and p2 in the collection
2. Replace theese two points with one weighted point p3 = (p1*m1+p2*m2)/(p1+p2)

There are some optimization to make histogram builder faster:

1. Spool: big map that saves from excessively merging of small bin. This map can contains up to maxSpoolSize points and accumulate weight from same points. For example, if spoolSize=100, binSize=10 and there are only 50 different points. it will be only 40 merges regardless how many points will be added.
2. Spool is organized as open-addressing primitive hash map where odd elements are points and event elements are values. Spool can not resize => when number of collisions became bigger than threashold or size became large that array_size/2 Spool is drained to bin
3. DistanceHolder - sorted collection of distances between points in Bin. It is used to find nearest points in constant time
4. Distances and Bin organized as sorted arrays. It reduces garbage collection pressure and allows to find elements in log(binSize) time via binary search
5. To use existing Arrays.binarySearch {point, values} in bin and {distance, left_point} pairs is packed in one long

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamingTombstoneHistogramBuilder.AddResult

Constructor Summary

Constructors

Constructor and Description
StreamingTombstoneHistogramBuilder(int maxBinSize, int maxSpoolSize, int roundSeconds)

Method Summary

Modifier and Type	Method and Description
TombstoneHistogram	build() Creates a 'finished' snapshot of the current state of the historgram, but leaves this builder instance open for subsequent additions to the histograms.
void	flushHistogram() Drain the temporary spool into the final bins
void	update(int p) Adds new point p to this histogram.
void	update(int p, int m) Adds new point p with value m to this histogram.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingTombstoneHistogramBuilder

public StreamingTombstoneHistogramBuilder(int maxBinSize,
                                          int maxSpoolSize,
                                          int roundSeconds)

Method Detail

update

public void update(int p)

Adds new point p to this histogram.

Parameters:

p -

update

public void update(int p,
                   int m)

Adds new point p with value m to this histogram.

Parameters:

p -

m -

flushHistogram

public void flushHistogram()

Drain the temporary spool into the final bins

build

public TombstoneHistogram build()

Creates a 'finished' snapshot of the current state of the historgram, but leaves this builder instance open for subsequent additions to the histograms. Basically, this allows us to have some degree of sanity wrt sstable early open.