JmsDataObject

Instance Constructors

new JmsDataObject(id: DataObjectId, jndiContextFactory: String, jndiProviderUrl: String, schemaMin: Option[StructType], authMode: AuthMode, batchSize: Int, maxWaitSec: Int, maxBatchAgeSec: Int, txBatchSize: Int, connectionFactory: String, queue: String, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

jndiContextFactory
JNDI Context Factory
jndiProviderUrl
JNDI Provider URL
authMode
authentication information: for now BasicAuthMode is supported.
batchSize
JMS batch size
connectionFactory
JMS Connection Factory
queue
Name of MQ Queue

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val authMode: AuthMode

authentication information: for now BasicAuthMode is supported.
val basicAuthMode: BasicAuthMode
val batchSize: Int

JMS batch size
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val connectionFactory: String

JMS Connection Factory
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def factory: FromConfigFactory[DataObject]

Returns the factory that can parse this type (that is, type CO).
Returns the factory that can parse this type (that is, type CO).
Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.
returns
the factory (object) for this class.

Definition Classes
JmsDataObject → ParsableFromConfig
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

Handle class cast exception when getting objects from instance registry
Handle class cast exception when getting objects from instance registry

Attributes
protected
Definition Classes
DataObject
def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

Attributes
protected
Definition Classes
DataObject
def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession): DataFrame

Definition Classes
JmsDataObject → CanCreateDataFrame
val id: DataObjectId

A unique identifier for this instance.
A unique identifier for this instance.

Definition Classes
JmsDataObject → DataObject → SdlConfigObject
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val jndiContextFactory: String

JNDI Context Factory
val jndiProviderUrl: String

JNDI Provider URL
lazy val logger: Logger

Attributes
protected
Definition Classes
SmartDataLakeLogger
val maxBatchAgeSec: Int
val maxWaitSec: Int
val metadata: Option[DataObjectMetadata]

Additional metadata for the DataObject
Additional metadata for the DataObject

Definition Classes
JmsDataObject → DataObject
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def postRead(implicit session: SparkSession): Unit

Runs operations after reading from DataObject
Runs operations after reading from DataObject

Definition Classes
DataObject
def postWrite(implicit session: SparkSession): Unit

Runs operations after writing to DataObject
Runs operations after writing to DataObject

Definition Classes
DataObject
def preRead(implicit session: SparkSession): Unit

Runs operations before reading from DataObject
Runs operations before reading from DataObject

Definition Classes
DataObject
def preWrite(implicit session: SparkSession): Unit

Runs operations before writing to DataObject
Runs operations before writing to DataObject

Definition Classes
DataObject
def prepare(implicit session: SparkSession): Unit

Prepare & test DataObject's prerequisits
Prepare & test DataObject's prerequisits
This runs during the "prepare" operation of the DAG.

Definition Classes
DataObject
val queue: String

Name of MQ Queue
val schemaMin: Option[StructType]

An optional, minimal schema that a DataObject schema must have to pass schema validation.
An optional, minimal schema that a DataObject schema must have to pass schema validation.
The schema validation semantics are: - Schema A is valid in respect to a minimal schema B when B is a subset of A. This means: the whole column set of B is contained in the column set of A.
- A column of B is contained in A when A contains a column with equal name and data type.
- Column order is ignored.
- Column nullability is ignored.
- Duplicate columns in terms of name and data type are eliminated (set semantics).
Note: This is only used by the functionality defined in CanCreateDataFrame and CanWriteDataFrame, that is, when reading or writing Spark data frames from/to the underlying data container. io.smartdatalake.workflow.action.Actions that bypass Spark data frames ignore the schemaMin attribute if it is defined.
Definition Classes
JmsDataObject → SchemaValidation
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toStringShort: String

Definition Classes
DataObject
val txBatchSize: Int
def validateSchemaMin(df: DataFrame): Unit

Validate the schema of a given Spark Data Frame df against schemaMin.
Validate the schema of a given Spark Data Frame df against schemaMin.
df
The data frame to validate.

Definition Classes
SchemaValidation
Exceptions thrown
SchemaViolationException is the schemaMin does not validate.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object JmsDataObject | package dataobject

Instance Constructors

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val authMode: AuthMode

val basicAuthMode: BasicAuthMode

val batchSize: Int

def clone(): AnyRef

val connectionFactory: String

final def eq(arg0: AnyRef): Boolean

def factory: FromConfigFactory[DataObject]

def finalize(): Unit

final def getClass(): Class[_]

def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession): DataFrame

val id: DataObjectId

final def isInstanceOf[T0]: Boolean

val jndiContextFactory: String

val jndiProviderUrl: String

lazy val logger: Logger

val maxBatchAgeSec: Int

val maxWaitSec: Int

val metadata: Option[DataObjectMetadata]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def postRead(implicit session: SparkSession): Unit

def postWrite(implicit session: SparkSession): Unit

def preRead(implicit session: SparkSession): Unit

def preWrite(implicit session: SparkSession): Unit

def prepare(implicit session: SparkSession): Unit

val queue: String

val schemaMin: Option[StructType]

final def synchronized[T0](arg0: ⇒ T0): T0

def toStringShort: String

val txBatchSize: Int

def validateSchemaMin(df: DataFrame): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SchemaValidation

Inherited from CanCreateDataFrame

Inherited from DataObject

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[DataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped