ActionsExporterDataObject

Exports a util DataFrame that contains properties and metadata extracted from all io.smartdatalake.workflow.action.Actions that are registered in the current InstanceRegistry.

Alternatively, it can export the properties and metadata of all io.smartdatalake.workflow.action.Actions defined in config files. For this, the configuration "config" has to be set to the location of the config.

Example:

dataObjects = {
 ...
 actions-exporter {
   type = ActionsExporterDataObject
   config = path/to/myconfiguration.conf
 }
 ...
}

The config value can point to a configuration file or a directory containing configuration files.

See also: Refer to ConfigLoader.loadConfigFromFilesystem() for details about the configuration loading.

Linear Supertypes

Serializable, Serializable, Product, Equals, CanCreateDataFrame, DataObject, AtlasExportable, SmartDataLakeLogger, ParsableFromConfig[ActionsExporterDataObject], SdlConfigObject, AnyRef, Any

Instance Constructors

new ActionsExporterDataObject(id: DataObjectId, config: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def addFieldIfNotExisting(writeSchema: StructType, colName: String, dataType: DataType): StructType

Attributes
protected
Definition Classes
CanCreateDataFrame
final def asInstanceOf[T0]: T0

Definition Classes
Any
def atlasName: String

Definition Classes
DataObject → AtlasExportable
def atlasQualifiedName(prefix: String): String

Definition Classes
AtlasExportable
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val config: Option[String]
def createReadSchema(writeSchema: StructType)(implicit session: SparkSession): StructType

Creates the read schema based on a given write schema.
Creates the read schema based on a given write schema. Normally this is the same, but some DataObjects can remove & add columns on read (e.g. KafkaTopicDataObject, SparkFileDataObject) In this cases we have to break the DataFrame lineage und create a dummy DataFrame in init phase.

Definition Classes
CanCreateDataFrame
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def factory: FromConfigFactory[ActionsExporterDataObject]

Returns the factory that can parse this type (that is, type CO).
Returns the factory that can parse this type (that is, type CO).
Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.
returns
the factory (object) for this class.

Definition Classes
ActionsExporterDataObject → ParsableFromConfig
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

Handle class cast exception when getting objects from instance registry
Handle class cast exception when getting objects from instance registry

Attributes
protected
Definition Classes
DataObject
def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

Attributes
protected
Definition Classes
DataObject
def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession, context: ActionPipelineContext): DataFrame

session
SparkSession to use
returns
DataFrame including all Actions in the instanceRegistry, used for exporting the metadata

Definition Classes
ActionsExporterDataObject → CanCreateDataFrame
val id: DataObjectId

A unique identifier for this instance.
A unique identifier for this instance.

Definition Classes
ActionsExporterDataObject → DataObject → SdlConfigObject
implicit val instanceRegistry: InstanceRegistry
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
SmartDataLakeLogger
val metadata: Option[DataObjectMetadata]

Additional metadata for the DataObject
Additional metadata for the DataObject

Definition Classes
ActionsExporterDataObject → DataObject
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toStringShort: String

Definition Classes
DataObject
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object ActionsExporterDataObject | package dataobject

Instance Constructors

new ActionsExporterDataObject(id: DataObjectId, config: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def addFieldIfNotExisting(writeSchema: StructType, colName: String, dataType: DataType): StructType

final def asInstanceOf[T0]: T0

def atlasName: String

def atlasQualifiedName(prefix: String): String

def clone(): AnyRef

val config: Option[String]

def createReadSchema(writeSchema: StructType)(implicit session: SparkSession): StructType

final def eq(arg0: AnyRef): Boolean

def factory: FromConfigFactory[ActionsExporterDataObject]

def finalize(): Unit

final def getClass(): Class[_]

def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession, context: ActionPipelineContext): DataFrame

val id: DataObjectId

implicit val instanceRegistry: InstanceRegistry

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

val metadata: Option[DataObjectMetadata]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toStringShort: String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from CanCreateDataFrame

Inherited from DataObject

Inherited from AtlasExportable

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[ActionsExporterDataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped