com.twitter.summingbird.scalding.store
These functions convert back and forth between a specific BatchID and the earliest time of the BatchID just after it.
These functions convert back and forth between a specific BatchID and the earliest time of the BatchID just after it.
The version numbers are the exclusive upper-bound of time covered by this store, while the batchIDs are the inclusive upper bound. Put another way, all events that occured before the version are included in this store.
The batcher for this store
The batcher for this store
Override this to control when keys are frozen.
Override this to control when keys are frozen. This allows us to avoid sorting and shuffling keys that are not updated.
instances of this trait MAY NOT change the logic here.
instances of this trait MAY NOT change the logic here. This always follows the rule that we look for existing data (avoiding reading deltas in that case), then we fall back to the last checkpointed output by calling readLast. In that case, we compute the results by rolling forward
For each batch, collect up values with the same key on mapside before the keys are expanded.
For each batch, collect up values with the same key on mapside before the keys are expanded.
Override this to set up store pruning, by default, no (key,value) pairs are pruned.
Override this to set up store pruning, by default, no (key,value) pairs are pruned. This is a house keeping function to permanently remove entries matching a criteria.
Returns a snapshot of the store's (K, V) pairs aggregated up to (but not including!) the time covered by the supplied batchID.
Returns a snapshot of the store's (K, V) pairs aggregated up to (but not including!) the time covered by the supplied batchID.
Aggregating the readLast for a particular batchID with the stream stored for the same batchID will return the aggregate up to (but not including) batchID.next. Streams deal with inclusive upper bound.
Returns a FlowProducer that supplies all data for the given specific version within this store's rootPath.
Returns a FlowProducer that supplies all data for the given specific version within this store's rootPath.
Make sure not to keep more than versionsToKeep when we write out.
Make sure not to keep more than versionsToKeep when we write out. If this is out of sync with VersionedKeyValSource we can have issues
For (firstNonZero - 1) we read empty.
For (firstNonZero - 1) we read empty. For all before we error on read. For all later, we proxy On write, we throw if batchID is less than firstNonZero
writeLast receives an INCLUSIVE upper bound on batchID and a pipe of all key-value pairs aggregated up to (and including) that batchID.
writeLast receives an INCLUSIVE upper bound on batchID and a pipe of all key-value pairs aggregated up to (and including) that batchID. (Yes, this is confusing, since a number of other methods talk about the EXCLUSIVE upper bound.)
This implementation of writeLast sinks all key-value pairs out into a VersionedStore directory whose tagged version is the EXCLUSIVE upper bound on batchID, or "batchID.next".