WriteAsNamedFilesAction

Instance Constructors

new WriteAsNamedFilesAction(label: String, tempBasePath: Path, destBasePath: Path, numberOfFiles: Int, filenamePrefix: String, format: String, options: Map[String, String])

label
Label to write
tempBasePath
Base location of temporary folder
destBasePath
Destination path to put files in
numberOfFiles
Number of files to generate
filenamePrefix
Prefix of name of the file up to the filenumber and extension
format
Format to write (e.g. parquet, csv)
options
Options to pass to the DataFrameWriter

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def actionName: String

For representing the action
For representing the action

Definition Classes
DataFlowAction
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def description: String

Definition Classes
DataFlowAction
val destBasePath: Path

Destination path to put files in
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
val filenamePrefix: String

Prefix of name of the file up to the filenumber and extension
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flowState(inputs: DataFlowEntities): DataFlowActionState

Action has the responsibility of assessing itself and produce DataFlowActionState, that will be used by the executors to determine if they can call performAction or not.
Action has the responsibility of assessing itself and produce DataFlowActionState, that will be used by the executors to determine if they can call performAction or not. Also can be used for progress monitoring. This will allow for more custom actions without modifying the executors
inputs
- action will study the state of the inputs in order to generate self assessment
returns
- an instance of the DataFlowActionState

Definition Classes
DataFlowAction
val format: String

Format to write (e.g.
Format to write (e.g. parquet, csv)
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val guid: String

Unique id of the action, but using it for adding behaviours can be problematic due to Interceptors that are defined at a much later stage.
Unique id of the action, but using it for adding behaviours can be problematic due to Interceptors that are defined at a much later stage. Because of that ActionSchedulers must NOT use this guid.

Definition Classes
DataFlowAction
val inputLabels: List[String]

The unique identifiers for the inputs to this action
The unique identifiers for the inputs to this action

Definition Classes
WriteAsNamedFilesAction → DataFlowAction
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val label: String

Label to write
def logLabel: String

Definition Classes
DataFlowAction
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val numberOfFiles: Int

Number of files to generate
val options: Map[String, String]

Options to pass to the DataFrameWriter
val outputLabels: List[String]

The unique identifiers for the outputs to this action
The unique identifiers for the outputs to this action

Definition Classes
WriteAsNamedFilesAction → DataFlowAction
def performAction(inputs: DataFlowEntities, flowContext: SparkFlowContext): Try[ActionResult]

Definition Classes
WriteAsNamedFilesAction → SparkDataFlowAction
def performAction[C <: FlowContext](inputs: DataFlowEntities, flowContext: C): Try[ActionResult]

Perform the action
Perform the action
inputs
the DataFlowEntities corresponding to the inputLabels
flowContext
context of the flow in which this action runs
returns
the action outputs (these must be declared in the same order as their labels in outputLabels)

Definition Classes
SparkDataFlowAction → DataFlowAction
def requiresAllInputs: Boolean

This action can only be executed if all of the inputs are not empty.
This action can only be executed if all of the inputs are not empty. An input can be explicitly marked as empty. If false, than one or more inputs can be empty to start execution.

Definition Classes
DataFlowAction
def schedulingGuid: String

Interceptors must not override this property, as certain behaviours of the data flow (Ex, execution pools) will be associated with this scheduling guid.
Interceptors must not override this property, as certain behaviours of the data flow (Ex, execution pools) will be associated with this scheduling guid. Also ActionScheduler will use this guid to track scheduled actions.

Definition Classes
DataFlowAction
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val tempBasePath: Path

Base location of temporary folder
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object WriteAsNamedFilesAction | package spark

case class WriteAsNamedFilesAction(label: String, tempBasePath: Path, destBasePath: Path, numberOfFiles: Int, filenamePrefix: String, format: String, options: Map[String, String]) extends SparkDataFlowAction with Product with Serializable

Instance Constructors

new WriteAsNamedFilesAction(label: String, tempBasePath: Path, destBasePath: Path, numberOfFiles: Int, filenamePrefix: String, format: String, options: Map[String, String])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def actionName: String

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def description: String

val destBasePath: Path

final def eq(arg0: AnyRef): Boolean

val filenamePrefix: String

def finalize(): Unit

def flowState(inputs: DataFlowEntities): DataFlowActionState

val format: String

final def getClass(): Class[_]

val guid: String

val inputLabels: List[String]

final def isInstanceOf[T0]: Boolean

val label: String

def logLabel: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val numberOfFiles: Int

val options: Map[String, String]

val outputLabels: List[String]

def performAction(inputs: DataFlowEntities, flowContext: SparkFlowContext): Try[ActionResult]

def performAction[C <: FlowContext](inputs: DataFlowEntities, flowContext: C): Try[ActionResult]

def requiresAllInputs: Boolean

def schedulingGuid: String

final def synchronized[T0](arg0: ⇒ T0): T0

val tempBasePath: Path

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SparkDataFlowAction

Inherited from DataFlowAction

Inherited from AnyRef

Inherited from Any

Ungrouped