SparkSubFeedAction

Instance Constructors

new SparkSubFeedAction()

Abstract Value Members

abstract def breakDataFrameLineage: Boolean

Stop propagating input DataFrame through action and instead get a new DataFrame from DataObject.
Stop propagating input DataFrame through action and instead get a new DataFrame from DataObject. This can help to save memory and performance if the input DataFrame includes many transformations from previous Actions. The new DataFrame will be initialized according to the SubFeed's partitionValues.
abstract def factory: FromConfigFactory[Action]

Returns the factory that can parse this type (that is, type CO).
Returns the factory that can parse this type (that is, type CO).
Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.
returns
the factory (object) for this class.

Definition Classes
ParsableFromConfig
abstract val id: ActionObjectId

A unique identifier for this instance.
A unique identifier for this instance.

Definition Classes
Action → SdlConfigObject
abstract def initExecutionMode: Option[ExecutionMode]

Execution mode if this Action is a start node of a DAG run
abstract def input: DataObject with CanCreateDataFrame

Input DataObject which can CanCreateDataFrame
abstract def inputs: Seq[DataObject]

Input DataObjects To be implemented by subclasses
Input DataObjects To be implemented by subclasses

Definition Classes
Action
abstract def metadata: Option[ActionMetadata]

Additional metadata for the Action
Additional metadata for the Action

Definition Classes
Action
abstract def output: DataObject with CanWriteDataFrame

Output DataObject which can CanWriteDataFrame
abstract def outputs: Seq[DataObject]

Output DataObjects To be implemented by subclasses
Output DataObjects To be implemented by subclasses

Definition Classes
Action
abstract def persist: Boolean

Force persisting DataFrame on Disk.
Force persisting DataFrame on Disk. This helps to reduce memory needed for caching the DataFrame content and can serve as a recovery point in case an task get's lost.
abstract def transform(subFeed: SparkSubFeed)(implicit session: SparkSession, context: ActionPipelineContext): SparkSubFeed

Transform a SparkSubFeed.
Transform a SparkSubFeed. To be implemented by subclasses.
subFeed
SparkSubFeed to be transformed
returns
transformed SparkSubFeed

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def addRuntimeEvent(phase: String, state: RuntimeEventState, msg: String): Unit

Adds an action event
Adds an action event

Definition Classes
Action
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def exec(subFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Seq[SubFeed]

Action.exec implementation
Action.exec implementation
subFeeds
SparkSubFeed's to be processed
returns
processed SparkSubFeed's

Definition Classes
SparkSubFeedAction → Action
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getInputDataObject[T <: DataObject](id: DataObjectId)(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.TypeTag[T], registry: InstanceRegistry): T

Attributes
protected
Definition Classes
Action
def getOutputDataObject[T <: DataObject](id: DataObjectId)(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.TypeTag[T], registry: InstanceRegistry): T

Attributes
protected
Definition Classes
Action
def getRuntimeState: Option[String]

Definition Classes
Action
def hashCode(): Int

Definition Classes
AnyRef → Any
final def init(subFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Seq[SubFeed]

Action.init implementation
Action.init implementation
subFeeds
SparkSubFeed's to be processed
returns
processed SparkSubFeed's

Definition Classes
SparkSubFeedAction → Action
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
SmartDataLakeLogger
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def nodeId: String

provide an implementation of the DAG node id
provide an implementation of the DAG node id

Definition Classes
Action → DAGNode
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def postExec(inputSubFeeds: Seq[SubFeed], outputSubFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Unit

Executes operations needed after executing an action.
Executes operations needed after executing an action. In this step any operation on Input- or Output-DataObjects needed after the main task is executed, e.g. JdbcTableDataObjects postSql or CopyActions deleteInputData.

Definition Classes
SparkSubFeedAction → Action
def postExecSubFeed(inputSubFeed: SubFeed, outputSubFeed: SubFeed)(implicit session: SparkSession, context: ActionPipelineContext): Unit
def preExec(implicit session: SparkSession, context: ActionPipelineContext): Unit

Executes operations needed before executing an action.
Executes operations needed before executing an action. In this step any operation on Input- or Output-DataObjects needed before the main task is executed, e.g. JdbcTableDataObjects preSql

Definition Classes
Action
def prepare(implicit session: SparkSession, context: ActionPipelineContext): Unit

Prepare DataObjects prerequisites.
Prepare DataObjects prerequisites. In this step preconditions are prepared & tested: - directories exists or can be created - connections can be created
This runs during the "prepare" operation of the DAG.

Definition Classes
Action
def setSparkJobDescription(operation: String)(implicit session: SparkSession): Unit

Sets the util job description for better traceability in the Spark UI
Sets the util job description for better traceability in the Spark UI
operation
operation description (be short...)
session
util session

Definition Classes
Action
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
final def toString(): String

This is displayed in ascii graph visualization
This is displayed in ascii graph visualization

Definition Classes
Action → AnyRef → Any
def toStringMedium: String

Definition Classes
Action
def toStringShort: String

Definition Classes
Action
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package action

abstract class SparkSubFeedAction extends Action

Instance Constructors

new SparkSubFeedAction()

Abstract Value Members

abstract def breakDataFrameLineage: Boolean

abstract def factory: FromConfigFactory[Action]

abstract val id: ActionObjectId

abstract def initExecutionMode: Option[ExecutionMode]

abstract def input: DataObject with CanCreateDataFrame

abstract def inputs: Seq[DataObject]

abstract def metadata: Option[ActionMetadata]

abstract def output: DataObject with CanWriteDataFrame

abstract def outputs: Seq[DataObject]

abstract def persist: Boolean

abstract def transform(subFeed: SparkSubFeed)(implicit session: SparkSession, context: ActionPipelineContext): SparkSubFeed

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def addRuntimeEvent(phase: String, state: RuntimeEventState, msg: String): Unit

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

final def exec(subFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Seq[SubFeed]

def finalize(): Unit

final def getClass(): Class[_]

def getInputDataObject[T <: DataObject](id: DataObjectId)(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.TypeTag[T], registry: InstanceRegistry): T

def getOutputDataObject[T <: DataObject](id: DataObjectId)(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.TypeTag[T], registry: InstanceRegistry): T

def getRuntimeState: Option[String]

def hashCode(): Int

final def init(subFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Seq[SubFeed]

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

final def ne(arg0: AnyRef): Boolean

def nodeId: String

final def notify(): Unit

final def notifyAll(): Unit

final def postExec(inputSubFeeds: Seq[SubFeed], outputSubFeeds: Seq[SubFeed])(implicit session: SparkSession, context: ActionPipelineContext): Unit

def postExecSubFeed(inputSubFeed: SubFeed, outputSubFeed: SubFeed)(implicit session: SparkSession, context: ActionPipelineContext): Unit

def preExec(implicit session: SparkSession, context: ActionPipelineContext): Unit

def prepare(implicit session: SparkSession, context: ActionPipelineContext): Unit

def setSparkJobDescription(operation: String)(implicit session: SparkSession): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

final def toString(): String

def toStringMedium: String

def toStringShort: String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Action

Inherited from SmartDataLakeLogger

Inherited from DAGNode

Inherited from ParsableFromConfig[Action]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped