org.apache.spark.sql.catalyst.util
QuantileSummaries
Related Docs:
object QuantileSummaries
| package util
org.apache.spark.sql.catalyst.util
the compression threshold. After the internal buffer of statistics crosses this size, it attempts to compress the statistics together.
the target relative error. It is uniform across the complete range of values.
a buffer of quantile statistics. See the G-K article for more details.
the count of all the elements *inserted in the sampled buffer* (excluding the head buffer)
Returns a new summary that compresses the summary statistics and the head buffer.
Returns a new summary that compresses the summary statistics and the head buffer.
This implements the COMPRESS function of the GK algorithm. It does not modify the object.
a new summary object with compressed statistics
the compression threshold.
the compression threshold. After the internal buffer of statistics crosses this size, it attempts to compress the statistics together.
the count of all the elements *inserted in the sampled buffer* (excluding the head buffer)
Returns a summary with the given observation inserted into the summary.
Returns a summary with the given observation inserted into the summary. This method may either modify in place the current summary (and return the same summary, modified in place), or it may create a new summary from scratch it necessary.
the new observation to insert into the summary
Merges two (compressed) summaries together.
Merges two (compressed) summaries together.
Returns a new summary.
Runs a query for a given quantile.
Runs a query for a given quantile. The result follows the approximation guarantees detailed above. The query can only be run on a compressed summary: you need to call compress() before using it.
the target quantile
the target relative error.
the target relative error. It is uniform across the complete range of values.
a buffer of quantile statistics.
a buffer of quantile statistics. See the G-K article for more details.
Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (http://dx.doi.org/10.1145/375663.375670)
In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.