Packages

t

org.apache.spark.sql.execution.datasources

WriteJobStatsTracker

trait WriteJobStatsTracker extends Serializable

A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) WriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g. save them to memory/disk, issue warnings, etc). It is therefore important that such an objects is Serializable, as it will be sent from the driver to all executors.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Known Subclasses
BasicWriteJobStatsTracker
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WriteJobStatsTracker
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def newTaskInstance(): WriteTaskStatsTracker

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class.

    Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class. To be called by executors.

    returns

    A WriteTaskStatsTracker instance to be used for computing stats during a write task

  2. abstract def processStats(stats: Seq[WriteTaskStats]): Unit

    Process the given collection of stats computed during this job.

    Process the given collection of stats computed during this job. E.g. aggregate them, write them to memory / disk, issue warnings, whatever.

    stats

    One WriteTaskStats object from each successful write task.

    Note

    The type of @param stats is too generic. These classes should probably be parametrized: WriteTaskStatsTracker[S <: WriteTaskStats] WriteJobStatsTracker[S <: WriteTaskStats, T <: WriteTaskStatsTracker[S]] and this would then be: def processStats(stats: Seq[S]): Unit but then we wouldn't be able to have a Seq[WriteJobStatsTracker] due to type co-/contra-variance considerations. Instead, you may feel free to just cast stats to the expected derived type when implementing this method in a derived class. The framework will make sure to call this with the right arguments.