Packages

abstract class OneToManyTTLState extends TTLState

OneToManyTTLState is an implementation of TTLState for stateful variables that associate a single key with multiple values; every value has its own expiration timestamp.

We need an efficient way to find all the values that have expired, but we cannot issue point-wise deletes to the elements, since they are merged together using the RocksDB StringAppendOperator for merging. As such, we cannot keep a secondary index on the key (expirationMs, groupingKey, indexInList), since we have no way to delete a specific indexInList from the RocksDB value. (In the future, we could write a custom merge operator that can handle tombstones for deleted indexes, but RocksDB doesn't support custom merge operators written in Java/Scala.)

Instead, we manage expiration per grouping key instead. Our secondary index will look like (expirationMs, groupingKey) -> EMPTY_ROW. This way, we can quickly find all the grouping keys that contain at least one element that has expired.

To make sure that we aren't "late" in cleaning up expired values, this secondary index maps from the minimum expiration in a list and a grouping key to the EMPTY_VALUE. This index is called the "TTL index" in the code (to be consistent with OneToOneTTLState), though it behaves more like a work queue of lists that need to be cleaned up.

Since a grouping key may have a large list and we need to quickly know what the minimum expiration is, we need to reverse this work queue index. This reversed index maps from key to the minimum expiration in the list, and it is called the "min-expiry" index.

Note: currently, this is only used by ListState with TTL.

Linear Supertypes
TTLState, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OneToManyTTLState
  2. TTLState
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new OneToManyTTLState(stateNameArg: String, storeArg: StateStore, elementKeySchemaArg: StructType, ttlConfigArg: TTLConfig, batchTimestampMsArg: Long, metricsArg: Map[String, SQLMetric])

Type Members

  1. case class ValueExpirationResult(numValuesExpired: Long, newMinExpirationMs: Option[Long]) extends Product with Serializable

Abstract Value Members

  1. abstract def clearExpiredValues(elementKey: UnsafeRow): ValueExpirationResult
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  15. def toString(): String
    Definition Classes
    AnyRef → Any
  16. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  17. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  18. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from TTLState

Inherited from AnyRef

Inherited from Any

Ungrouped