com.twitter.summingbird.scalding

VersionedBatchStore

class VersionedBatchStore[K, V, K2, V2] extends VersionedBatchStoreBase[K, V]

Linear Supertypes
VersionedBatchStoreBase[K, V], BatchedScaldingStore[K, V], ScaldingStore[K, V], Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. VersionedBatchStore
  2. VersionedBatchStoreBase
  3. BatchedScaldingStore
  4. ScaldingStore
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new VersionedBatchStore(rootPath: String, versionsToKeep: Int, batcher: Batcher)(pack: (BatchID, (K, V)) ⇒ (K2, V2))(unpack: ((K2, V2)) ⇒ (K, V))(implicit injection: Injection[(K2, V2), (Array[Byte], Array[Byte])], ordering: Ordering[K])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def batchIDToVersion(b: BatchID): Long

    These functions convert back and forth between a specific BatchID and the earliest time of the BatchID just after it.

    These functions convert back and forth between a specific BatchID and the earliest time of the BatchID just after it.

    The version numbers are the exclusive upper-bound of time covered by this store, while the batchIDs are the inclusive upper bound. Put another way, all events that occured before the version are included in this store.

    Definition Classes
    VersionedBatchStoreBase
  8. val batcher: Batcher

    The batcher for this store

    The batcher for this store

    Definition Classes
    VersionedBatchStoreBatchedScaldingStore
  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  16. def lastBatch(exclusiveUB: BatchID, mode: Hdfs): Option[(BatchID, FlowProducer[TypedPipe[(K, V)]])]

    Attributes
    protected
    Definition Classes
    VersionedBatchStoreBase
  17. final def merge(delta: PipeFactory[(K, V)], sg: Semigroup[V], commutativity: Commutativity, reducers: Int): PipeFactory[(K, (Option[V], V))]

    instances of this trait MAY NOT change the logic here.

    instances of this trait MAY NOT change the logic here. This always follows the rule that we look for existing data (avoiding reading deltas in that case), then we fall back to the last checkpointed output by calling readLast. In that case, we compute the results by rolling forward

    Definition Classes
    BatchedScaldingStoreScaldingStore
  18. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  19. final def notify(): Unit

    Definition Classes
    AnyRef
  20. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  21. implicit val ordering: Ordering[K]

  22. def partialMerge[K1](delta: PipeFactory[(K1, V)], sg: Semigroup[V], commutativity: Commutativity): PipeFactory[(K1, V)]

    For each batch, collect up values with the same key on mapside before the keys are expanded.

    For each batch, collect up values with the same key on mapside before the keys are expanded.

    Definition Classes
    BatchedScaldingStoreScaldingStore
  23. def readLast(exclusiveUB: BatchID, mode: Mode): Try[(BatchID, FlowProducer[TypedPipe[(K, V)]])]

    Returns a snapshot of the store's (K, V) pairs aggregated up to (but not including!) the time covered by the supplied batchID.

    Returns a snapshot of the store's (K, V) pairs aggregated up to (but not including!) the time covered by the supplied batchID.

    Aggregating the readLast for a particular batchID with the stream stored for the same batchID will return the aggregate up to (but not including) batchID.next. Streams deal with inclusive upper bound.

    Definition Classes
    VersionedBatchStoreBaseBatchedScaldingStore
  24. def readVersion(v: Long): FlowProducer[TypedPipe[(K, V)]]

    Returns a FlowProducer that supplies all data for the given specific version within this store's rootPath.

    Returns a FlowProducer that supplies all data for the given specific version within this store's rootPath.

    Attributes
    protected
    Definition Classes
    VersionedBatchStoreVersionedBatchStoreBase
  25. def select(b: List[BatchID]): List[BatchID]

    Make sure not to keep more than versionsToKeep when we write out.

    Make sure not to keep more than versionsToKeep when we write out. If this is out of sync with VersionedKeyValSource we can have issues

    Definition Classes
    VersionedBatchStoreBatchedScaldingStore
  26. def sumByBatches[K1, V](ins: TypedPipe[(Long, (K1, V))], capturedBatcher: Batcher, commutativity: Commutativity)(implicit arg0: Semigroup[V]): TypedPipe[((K1, BatchID), (Timestamp, V))]

    Attributes
    protected
    Definition Classes
    BatchedScaldingStore
  27. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  28. def toString(): String

    Definition Classes
    AnyRef → Any
  29. def versionToBatchID(ver: Long): BatchID

    Definition Classes
    VersionedBatchStoreBase
  30. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. def withInitialBatch(firstNonZero: BatchID): BatchedScaldingStore[K, V]

    For (firstNonZero - 1) we read empty.

    For (firstNonZero - 1) we read empty. For all before we error on read. For all later, we proxy On write, we throw if batchID is less than firstNonZero

    Definition Classes
    BatchedScaldingStore
  34. def writeLast(batchID: BatchID, lastVals: TypedPipe[(K, V)])(implicit flowDef: FlowDef, mode: Mode): Unit

    writeLast receives an INCLUSIVE upper bound on batchID and a pipe of all key-value pairs aggregated up to (and including) that batchID.

    writeLast receives an INCLUSIVE upper bound on batchID and a pipe of all key-value pairs aggregated up to (and including) that batchID. (Yes, this is confusing, since a number of other methods talk about the EXCLUSIVE upper bound.)

    This implementation of writeLast sinks all key-value pairs out into a VersionedStore directory whose tagged version is the EXCLUSIVE upper bound on batchID, or "batchID.next".

    Definition Classes
    VersionedBatchStoreBatchedScaldingStore

Inherited from VersionedBatchStoreBase[K, V]

Inherited from BatchedScaldingStore[K, V]

Inherited from ScaldingStore[K, V]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped