com.twitter.finatra.streams.stores
internal
A write-behind caching layer around the FinatraKeyValueStore.
This class stores a global list of all Finatra Key Value Stores for use by Finatra's queryable state functionality.
This class stores a global list of all Finatra Key Value Stores for use by Finatra's queryable state functionality.
Note: We maintain or own global list of state stores so we can retrieve Finatra FinatraKeyValueStore implementations directly for querying (the alternative would be to retrieve the underlying RocksDBStore which is wrapped by the FinatraKeyValueStore implementations).
A write-behind caching layer around the FinatraKeyValueStore.
We cache Java objects here and then periodically flush entries into RocksDB which involves serializing the objects into byte arrays. As such this cache: 1) Reduces the number of reads/writes to RocksDB 2) Reduces the number of serialization/deserialization operations which can be expensive for some classes 3) Reduces the number of publishes to the Kafka changelog topic backing this key value store
This caching does introduce a few odd corner cases :-( 1. Items in the cache have pass-by-reference semantics but items in rocksdb have pass-by-value semantics. Modifying items after a put is a bad idea! Ideally, only immutable objects would be stored in a CachingFinatraKeyValueStore 2. Range queries currently only work against the uncached RocksDB data. This is because sorted Java maps are much less performant than their unsorted counterparts. We typically only use range queries for queryable state where it is ok to read stale data If fresher data is required for range queries, decrease your commit interval.
This class is inspired by: https://github.com/apache/samza/blob/1.0.0/samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala