Object

za.co.absa.atum.utils.controlmeasure

ControlMeasureUtils

Related Doc: package controlmeasure

Permalink

object ControlMeasureUtils

This object contains utilities used in Control Measurements processing

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ControlMeasureUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. object JsonType extends Enumeration

    Permalink
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def convertControlValuesToStrings(controlMeasure: ControlMeasure): ControlMeasure

    Permalink

    Converts all measurements in an instance of ControlMeasure object into stings so it won't cause confusion when deserialized downstream.

    Converts all measurements in an instance of ControlMeasure object into stings so it won't cause confusion when deserialized downstream.

    controlMeasure

    A control measures.

    returns

    The converted control measurements.

  8. val dateFormat: DateTimeFormatter

    Permalink
  9. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  13. def getTemporaryColumnName(df: DataFrame, namePrefix: String = "tmp"): String

    Permalink

    The method generates a temporary column name which does not exist in the specified DataFrame.

    The method generates a temporary column name which does not exist in the specified DataFrame.

    returns

    An column name as a string

  14. def getTimestampAsString: String

    Permalink

    Get current time as a string formatted according to Control Framework format za.co.absa.atum.utils.controlmeasure.ControlMeasureUtils#timestampFormat.

    Get current time as a string formatted according to Control Framework format za.co.absa.atum.utils.controlmeasure.ControlMeasureUtils#timestampFormat.

    returns

    The current timestamp as a string (e.g. "05-10-2017 09:43:50 +0200")

  15. def getTodayAsString: String

    Permalink

    Get current date as a string formatted according to Control Framework format za.co.absa.atum.utils.controlmeasure.ControlMeasureUtils#dateFormat().

    Get current date as a string formatted according to Control Framework format za.co.absa.atum.utils.controlmeasure.ControlMeasureUtils#dateFormat().

    returns

    The current date as a string (e.g. "05-10-2017")

  16. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  17. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  19. def normalize(controlMeasure: ControlMeasure): ControlMeasure

    Permalink

    Normalizes all measurements in an instance of ControlMeasure object into standard values

    Normalizes all measurements in an instance of ControlMeasure object into standard values

    controlMeasure

    A control measures.

    returns

    The normalized control measurements.

  20. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  22. def preprocessControlMeasure: (ControlMeasure) ⇒ ControlMeasure

    Permalink
  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  24. val timestampFormat: DateTimeFormatter

    Permalink
  25. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  26. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. def writeControlMeasureInfoFileToHadoopFs(cm: ControlMeasure, outputDir: Path, jsonType: JsonType = JsonType.Minified)(implicit outputFs: FileSystem): Unit

    Permalink

    Will write Control Measure cm as JSON to Hadoop FS (by default to into the dir specified in cm.metadata.dataFileName, file name: _INFO)

    Will write Control Measure cm as JSON to Hadoop FS (by default to into the dir specified in cm.metadata.dataFileName, file name: _INFO)

    cm

    control measure

    outputDir

    dir on outputFs, usual choice is cm.metadata.dataFileName

    jsonType

    JsonType.Minified for compact json (no whitespaces) or JsonType.Pretty for indented

    outputFs

    hadoop FS. For regular HDFS, use e.g. FileSystem.get(sparkSession.sparkContext.hadoopConfiguration) or your S3 FS (or rely on e.g. org.apache.hadoop.conf.Configuration))

Deprecated Value Members

  1. def asJson[T <: AnyRef](obj: T): String

    Permalink

    The method returns arbitrary object as a Json string.

    The method returns arbitrary object as a Json string. Calls za.co.absa.atum.utils.SerializationUtils#asJson(java.lang.Object)

    returns

    A string representing the object in Json format

    Annotations
    @deprecated
    Deprecated

    (Since version 3.3.0) Use SerializationUtils.asJson instead

  2. def asJsonPretty[T <: AnyRef](obj: T): String

    Permalink

    The method returns arbitrary object as a pretty Json string.

    The method returns arbitrary object as a pretty Json string. Calls za.co.absa.atum.utils.SerializationUtils#asJsonPretty(java.lang.Object)

    returns

    A string representing the object in Json format

    Annotations
    @deprecated
    Deprecated

    (Since version 3.3.0) Use SerializationUtils.asJsonPretty instead

  3. def createInfoFile(ds: Dataset[Row], sourceApplication: String, inputPathName: String, reportDate: String = getTodayAsString, reportVersion: Int = 1, country: String = "ZA", historyType: String = "Snapshot", sourceType: String = "Source", initialCheckpointName: String = "Source", workflowName: String = "Source", writeToHDFS: Boolean = true, prettyJSON: Boolean = true, aggregateColumns: Seq[String]): String

    Permalink

    The method crates an _INFO file for a given dataset.

    The method crates an _INFO file for a given dataset. The row count measurement is added automatically. You can also specify aggregation columns for aggregation measurements

    ds

    A dataset for which _INFO file to be created.

    sourceApplication

    The name of the application providing the data.

    inputPathName

    The path to the input file name. Can be a folder with file mask.

    reportDate

    The date of the data generation (default = today).

    reportVersion

    The version of the data generation for the date, new versions replace old versions of data (default = 1).

    country

    Country name (default = "ZA").

    historyType

    History type (default = "Snapshot").

    sourceType

    Source type (default = "Source").

    initialCheckpointName

    The name of the initial checkpoint (default = "Source").

    workflowName

    A workflow name to group several checkpoint sth in the chain (default = "Source").

    writeToHDFS

    A flag specifying if saving _INFO file to HDFS needed. If false the _INFO file will not be saved to HDFS.

    prettyJSON

    Output pretty JSON.

    aggregateColumns

    Numeric column names for.

    returns

    The content of the _INFO file.

    Annotations
    @deprecated
    Deprecated

    (Since version 3.4.0) Use ControlMeasureBuilder.forDf(...) ... .build & ControlMeasureUtils.writeControlMeasureInfoFileToHadoopFs(...) instead

  4. def fromJson[T <: AnyRef](jsonStr: String)(implicit m: Manifest[T]): T

    Permalink

    The method returns arbitrary object parsed from Json string.

    The method returns arbitrary object parsed from Json string. Calls scala.reflect.Manifest)

    returns

    An object deserialized from the Json string

    Annotations
    @deprecated
    Deprecated

    (Since version 3.3.0) Use SerializationUtils.fromJson instead

Inherited from AnyRef

Inherited from Any

Ungrouped