Package org.apache.cassandra.metrics

Class DecayingEstimatedHistogramReservoir

  • All Implemented Interfaces:
    com.codahale.metrics.Reservoir, SnapshottingReservoir
    public class DecayingEstimatedHistogramReservoir
extends java.lang.Object
implements SnapshottingReservoir
    A decaying histogram reservoir where values collected during each minute will be twice as significant as the values collected in the previous minute. Measured values are collected in variable sized buckets, using small buckets in the lower range and larger buckets in the upper range. Use this histogram when you want to know if the distribution of the underlying data stream has changed recently and you want high resolution on values in the lower range.

    The histogram use forward decay [1] to make recent values more significant. The forward decay factor will be doubled every minute (half-life time set to 60 seconds) [2]. The forward decay landmark is reset every 30 minutes (or at first read/update after 30 minutes). During landmark reset, updates and reads in the reservoir will be blocked in a fashion similar to the one used in the metrics library [3]. The 30 minute rescale interval is used based on the assumption that in an extreme case we would have to collect a metric 1M times for a single bucket each second. By the end of the 30:th minute all collected values will roughly add up to 1.000.000 * 60 * pow(2, 30) which can be represented with 56 bits giving us some head room in a signed 64 bit long.

    Internally two reservoirs are maintained, one with decay and one without decay. All public getters in a Snapshot will expose the decay functionality with the exception of the Snapshot.getValues() which will return values from the reservoir without decay. This makes it possible for the caller to maintain precise deltas in an interval of its choice.

    The bucket size starts at 1 and grows by 1.2 each time (rounding and removing duplicates). It goes from 1 to around 18T by default (creating 164+1 buckets), which will give a timing resolution from microseconds to roughly 210 days, with less precision as the numbers get larger.

    The series of values to which the counts in `decayingBuckets` correspond: 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20, 24, 29, 35, 42, 50, 60, 72 etc. Thus, a `decayingBuckets` of [0, 0, 1, 10] would mean we had seen 1 value of 3 and 10 values of 4.

    Each bucket represents values from (previous bucket offset, current offset].

    To reduce contention each logical bucket is striped accross a configurable number of stripes (default: 2). Threads are assigned to specific stripes. In addition, logical buckets are distributed across the physical storage to reduce conention when logically adjacent buckets are updated. See CASSANDRA-15213.

    • [1]: http://dimacs.rutgers.edu/~graham/pubs/papers/fwddecay.pdf
    • [2]: https://en.wikipedia.org/wiki/Half-life
    • [3]: https://github.com/dropwizard/metrics/blob/v3.1.2/metrics-core/src/main/java/com/codahale/metrics/ExponentiallyDecayingReservoir.java

    • Field Detail

      • DEFAULT_BUCKET_COUNT

        public static final int DEFAULT_BUCKET_COUNT
        The default number of decayingBuckets. Use this bucket count to reduce memory allocation for bucket offsets.
        See Also:
        Constant Field Values

      • DEFAULT_STRIPE_COUNT

        public static final int DEFAULT_STRIPE_COUNT

      • DEFAULT_ZERO_CONSIDERATION

        public static final boolean DEFAULT_ZERO_CONSIDERATION
        See Also:
        Constant Field Values

      • DEFAULT_WITHOUT_ZERO_BUCKET_OFFSETS

        public static final long[] DEFAULT_WITHOUT_ZERO_BUCKET_OFFSETS

      • DEFAULT_WITH_ZERO_BUCKET_OFFSETS

        public static final long[] DEFAULT_WITH_ZERO_BUCKET_OFFSETS

      • MEAN_LIFETIME_IN_S

        public static final double MEAN_LIFETIME_IN_S

      • LANDMARK_RESET_INTERVAL_IN_NS

        public static final long LANDMARK_RESET_INTERVAL_IN_NS

    • Constructor Detail

      • DecayingEstimatedHistogramReservoir

        public DecayingEstimatedHistogramReservoir()
        Construct a decaying histogram with default number of buckets and without considering zeroes.

      • DecayingEstimatedHistogramReservoir

        public DecayingEstimatedHistogramReservoir​(boolean considerZeroes)
        Construct a decaying histogram with default number of buckets.
        Parameters:
        considerZeroes - when true, 0-value measurements in a separate bucket, otherwise they will be collected in same bucket as 1-value measurements

      • DecayingEstimatedHistogramReservoir

        public DecayingEstimatedHistogramReservoir​(boolean considerZeroes,
                                           int bucketCount,
                                           int stripes)
        Construct a decaying histogram.
        Parameters:
        considerZeroes - when true, 0-value measurements in a separate bucket, otherwise they will be collected in same bucket as 1-value measurements
        bucketCount - number of buckets used to collect measured values

      • DecayingEstimatedHistogramReservoir

        public DecayingEstimatedHistogramReservoir​(MonotonicClock clock)

    • Method Detail

      • fastLog12

        public static float fastLog12​(long v)

      • update

        public void update​(long value)
        Increments the count of the bucket closest to n, rounding UP.
        Specified by:
        update in interface com.codahale.metrics.Reservoir
        Parameters:
        value - the data point to add to the histogram

      • updateBucket

        public void updateBucket​(java.util.concurrent.atomic.AtomicLongArray buckets,
                         int index,
                         long value)

      • stripedIndex

        public int stripedIndex​(int offsetIndex,
                        int stripe)

      • findIndex

        public static int findIndex​(long[] bucketOffsets,
                            long value)

      • size

        public int size()
        Returns the logical number of buckets where recorded values are stored. The actual number of physical buckets is size() * stripeCount() This method does not return the number of recorded values as suggested by the Reservoir interface.
        Specified by:
        size in interface com.codahale.metrics.Reservoir
        Returns:
        the number of buckets
        See Also:
        stripeCount()

      • stripeCount

        public int stripeCount()

      • getSnapshot

        public com.codahale.metrics.Snapshot getSnapshot()
        Returns a snapshot of the decaying values in this reservoir. Non-decaying reservoir will not be included in the snapshot.
        Specified by:
        getSnapshot in interface com.codahale.metrics.Reservoir
        Returns:
        the snapshot

      • clear

        public void clear()

      • rebase

        public void rebase​(org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir.EstimatedHistogramReservoirSnapshot snapshot)
        Replaces current internal values with the given one from a Snapshot. This method is NOT thread safe, values added at the same time to this reservoir using methods such as update may lose their data