abstract class OneToManyTTLState extends TTLState
OneToManyTTLState is an implementation of TTLState for stateful variables that associate a single key with multiple values; every value has its own expiration timestamp.
We need an efficient way to find all the values that have expired, but we cannot issue point-wise deletes to the elements, since they are merged together using the RocksDB StringAppendOperator for merging. As such, we cannot keep a secondary index on the key (expirationMs, groupingKey, indexInList), since we have no way to delete a specific indexInList from the RocksDB value. (In the future, we could write a custom merge operator that can handle tombstones for deleted indexes, but RocksDB doesn't support custom merge operators written in Java/Scala.)
Instead, we manage expiration per grouping key instead. Our secondary index will look like (expirationMs, groupingKey) -> EMPTY_ROW. This way, we can quickly find all the grouping keys that contain at least one element that has expired.
To make sure that we aren't "late" in cleaning up expired values, this secondary index maps from the minimum expiration in a list and a grouping key to the EMPTY_VALUE. This index is called the "TTL index" in the code (to be consistent with OneToOneTTLState), though it behaves more like a work queue of lists that need to be cleaned up.
Since a grouping key may have a large list and we need to quickly know what the minimum expiration is, we need to reverse this work queue index. This reversed index maps from key to the minimum expiration in the list, and it is called the "min-expiry" index.
Note: currently, this is only used by ListState with TTL.
- Alphabetic
- By Inheritance
- OneToManyTTLState
- TTLState
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new OneToManyTTLState(stateNameArg: String, storeArg: StateStore, elementKeySchemaArg: StructType, ttlConfigArg: TTLConfig, batchTimestampMsArg: Long, metricsArg: Map[String, SQLMetric])
Type Members
- case class ValueExpirationResult(numValuesExpired: Long, newMinExpirationMs: Option[Long]) extends Product with Serializable
Abstract Value Members
- abstract def clearExpiredValues(elementKey: UnsafeRow): ValueExpirationResult
- Attributes
- protected
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)