Packages

class BasicWriteJobStatsTracker extends WriteJobStatsTracker

Simple WriteJobStatsTracker implementation that's serializable, capable of instantiating BasicWriteTaskStatsTracker on executors and processing the BasicWriteTaskStats they produce by aggregating the metrics and posting them as DriverMetricUpdates.

Linear Supertypes
WriteJobStatsTracker, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BasicWriteJobStatsTracker
  2. WriteJobStatsTracker
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BasicWriteJobStatsTracker(serializableHadoopConf: SerializableConfiguration, metrics: Map[String, SQLMetric])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. val metrics: Map[String, SQLMetric]
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def newTaskInstance(): WriteTaskStatsTracker

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class.

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class. To be called by executors.

    returns

    A WriteTaskStatsTracker instance to be used for computing stats during a write task

    Definition Classes
    BasicWriteJobStatsTracker → WriteJobStatsTracker
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def processStats(stats: Seq[WriteTaskStats]): Unit

    Process the given collection of stats computed during this job.

    Process the given collection of stats computed during this job. E.g. aggregate them, write them to memory / disk, issue warnings, whatever.

    stats

    One WriteTaskStats object from each successful write task.

    Definition Classes
    BasicWriteJobStatsTracker → WriteJobStatsTracker
    Note

    The type of @param stats is too generic. These classes should probably be parametrized: WriteTaskStatsTracker[S <: WriteTaskStats] WriteJobStatsTracker[S <: WriteTaskStats, T <: WriteTaskStatsTracker[S]] and this would then be: def processStats(stats: Seq[S]): Unit but then we wouldn't be able to have a Seq[WriteJobStatsTracker] due to type co-/contra-variance considerations. Instead, you may feel free to just cast stats to the expected derived type when implementing this method in a derived class. The framework will make sure to call this with the right arguments.

  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from WriteJobStatsTracker

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped