com.intenthq.pucket.writer

IncrementalPartitionedWriter

case class IncrementalPartitionedWriter[T] extends Writer[T, (Long, Throwable)] with PartitionedWriterFunctions[T, (Long, Throwable), IncrementalPartitionedWriter[T]] with Product with Serializable

Incremental partitioned parquet writer Provides incremental writing for partitioned puckets Any error will return with the minimum checkpoint of the current set of writers

Linear Supertypes
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. IncrementalPartitionedWriter
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. PartitionedWriterFunctions
  7. Writer
  8. AnyRef
  9. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. type Error = (Long, Throwable)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def checkPoint: Long

    Definition Classes
    Writer
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def close: \/[(Long, Throwable), Unit]

    Close the writer

    Close the writer

    returns

    an error if the close fails or unit

    Definition Classes
    IncrementalPartitionedWriterWriter
  10. def combineExceptions(last: Throwable, current: Throwable): Throwable

    Attributes
    protected
    Definition Classes
    Writer
  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  15. val maxWrites: Long

  16. def minCheckpoint: Long

    Used to find the lowest checkpoint for which data is confirmed to be written

    Used to find the lowest checkpoint for which data is confirmed to be written

    returns

    the lowest checkpoint

  17. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def newInstance(writers: Writers[T, (Long, Throwable)]): IncrementalPartitionedWriter[T]

    Create a new instance of the partitioned writer

    Create a new instance of the partitioned writer

    writers

    new writer cache state to be included in the new instance

    returns

    a new instance of a partitioned writer with the new state

    Definition Classes
    IncrementalPartitionedWriterPartitionedWriterFunctions
  19. def newWriter(partition: Path, checkPoint: Long): \/[(Long, Throwable), Writer[T, (Long, Throwable)]]

    Obtain a new writer for a partition

    Obtain a new writer for a partition

    partition

    the pucket instance for the partition

    checkPoint

    the current checkpoint

    returns

    a new writer for the partition or an error

    Definition Classes
    IncrementalPartitionedWriterPartitionedWriterFunctions
  20. final def notify(): Unit

    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  22. val pucket: Pucket[T]

  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  24. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. def write(data: T, checkPoint: Long): \/[(Long, Throwable), IncrementalPartitionedWriter[T]]

    Write data to a partition Finds a writer in the cache or creates a new one then submits the writer back to the cache

    Write data to a partition Finds a writer in the cache or creates a new one then submits the writer back to the cache

    data

    the data to be written

    checkPoint

    the current checkpoint to be passed to the underlying writer

    returns

    a new instance of the partitioned writer complete with new state

    Definition Classes
    PartitionedWriterFunctions
  28. def writerCache(partitionId: Path, writer: Writer[T, (Long, Throwable)]): \/[(Long, Throwable), Writers[T, (Long, Throwable)]]

    Add a new writer to the cache Will update an existing writer if one for the same partition already exists in cache.

    Add a new writer to the cache Will update an existing writer if one for the same partition already exists in cache.

    If a new writer needs to be cached and the cache is full, the oldest one in the cache will be closed and evicted.

    partitionId

    pucket identifier to use as a key in the cache

    writer

    the writer instance to be cached

    returns

    Definition Classes
    PartitionedWriterFunctions
  29. val writerCacheSize: Int

  30. val writers: Writers[T, (Long, Throwable)]

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from PartitionedWriterFunctions[T, (Long, Throwable), IncrementalPartitionedWriter[T]]

Inherited from Writer[T, (Long, Throwable)]

Inherited from AnyRef

Inherited from Any

Ungrouped