IncrementalWriter

Incremental functional writer class for parquet

Overview

Allows large files to be written to parquet in smaller incements. Closes the underlying writer and creates a new file at a defined writer count.

Also keeps a checkpoint of when the last file was sucessfully flushed to the filesystem so in the event of an error processing can be restarted from that point.

T: type of data to be written

Linear Supertypes

Serializable, Serializable, Product, Equals, Writer[T, (Long, Throwable)], AnyRef, Any

Type Members

type Error = (Long, Throwable)

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val checkPoint: Long

Definition Classes
IncrementalWriter → Writer
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def close: \/[(Long, Throwable), Unit]

Close the writer
Close the writer
returns
an error if the close fails or unit

Definition Classes
IncrementalWriter → Writer
def combineExceptions(last: Throwable, current: Throwable): Throwable

Attributes
protected
Definition Classes
Writer
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val maxWrites: Long
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val pucket: Pucket[T]
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def write(data: T, newCheckPoint: Long): \/[(Long, Throwable), IncrementalWriter[T]]

Write data incrementally
Write data incrementally
data
the data to be written
newCheckPoint
the current progress of processing in the subject file
returns
the writer instance or an error message with the checkpoint if the write fails

Definition Classes
IncrementalWriter → Writer
val writeCounter: Long
val writer: Writer[T, Throwable]

case class IncrementalWriter[T] extends Writer[T, (Long, Throwable)] with Product with Serializable

Overview

Type Members

type Error = (Long, Throwable)

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val checkPoint: Long

def clone(): AnyRef

def close: \/[(Long, Throwable), Unit]

def combineExceptions(last: Throwable, current: Throwable): Throwable

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

final def isInstanceOf[T0]: Boolean

val maxWrites: Long

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val pucket: Pucket[T]

final def synchronized[T0](arg0: ⇒ T0): T0

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def write(data: T, newCheckPoint: Long): \/[(Long, Throwable), IncrementalWriter[T]]

val writeCounter: Long

val writer: Writer[T, Throwable]

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Writer[T, (Long, Throwable)]

Inherited from AnyRef

Inherited from Any

Ungrouped