Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Definition Classes

  apache

•  package sql

  Definition Classes

  spark

•  package execution

  The physical execution component of Spark SQL.

  The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

  Definition Classes

  sql

•  package streaming

  Definition Classes

  execution

•  abstract class OneToManyTTLState extends TTLState

  OneToManyTTLState is an implementation of TTLState for stateful variables that associate a single key with multiple values; every value has its own expiration timestamp.

  OneToManyTTLState is an implementation of TTLState for stateful variables that associate a single key with multiple values; every value has its own expiration timestamp.

  We need an efficient way to find all the values that have expired, but we cannot issue point-wise deletes to the elements, since they are merged together using the RocksDB StringAppendOperator for merging. As such, we cannot keep a secondary index on the key (expirationMs, groupingKey, indexInList), since we have no way to delete a specific indexInList from the RocksDB value. (In the future, we could write a custom merge operator that can handle tombstones for deleted indexes, but RocksDB doesn't support custom merge operators written in Java/Scala.)

  Instead, we manage expiration per grouping key instead. Our secondary index will look like (expirationMs, groupingKey) -> EMPTY_ROW. This way, we can quickly find all the grouping keys that contain at least one element that has expired.

  To make sure that we aren't "late" in cleaning up expired values, this secondary index maps from the minimum expiration in a list and a grouping key to the EMPTY_VALUE. This index is called the "TTL index" in the code (to be consistent with OneToOneTTLState), though it behaves more like a work queue of lists that need to be cleaned up.

  Since a grouping key may have a large list and we need to quickly know what the minimum expiration is, we need to reverse this work queue index. This reversed index maps from key to the minimum expiration in the list, and it is called the "min-expiry" index.

  Note: currently, this is only used by ListState with TTL.

  Definition Classes

  streaming

• ValueExpirationResult

org.apache.spark.sql.execution.streaming.OneToManyTTLState

ValueExpirationResult