org.apache.spark.sql.execution.datasources
BasicWriteJobStatsTracker
Companion object BasicWriteJobStatsTracker
class BasicWriteJobStatsTracker extends WriteJobStatsTracker
Simple WriteJobStatsTracker implementation that's serializable, capable of instantiating BasicWriteTaskStatsTracker on executors and processing the BasicWriteTaskStats they produce by aggregating the metrics and posting them as DriverMetricUpdates.
- Alphabetic
- By Inheritance
- BasicWriteJobStatsTracker
- WriteJobStatsTracker
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- val driverSideMetrics: Map[String, SQLMetric]
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def newTaskInstance(): WriteTaskStatsTracker
Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class.
Instantiates a WriteTaskStatsTracker, based on (non-transient) members of this class. To be called by executors.
- returns
A WriteTaskStatsTracker instance to be used for computing stats during a write task
- Definition Classes
- BasicWriteJobStatsTracker → WriteJobStatsTracker
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def processStats(stats: Seq[WriteTaskStats], jobCommitTime: Long): Unit
Process the given collection of stats computed during this job.
Process the given collection of stats computed during this job. E.g. aggregate them, write them to memory / disk, issue warnings, whatever.
- stats
One WriteTaskStats object from each successful write task.
- jobCommitTime
Time of committing the job.
- Definition Classes
- BasicWriteJobStatsTracker → WriteJobStatsTracker
- Note
The type of @param
statsis too generic. These classes should probably be parametrized: WriteTaskStatsTracker[S <: WriteTaskStats] WriteJobStatsTracker[S <: WriteTaskStats, T <: WriteTaskStatsTracker[S]] and this would then be: def processStats(stats: Seq[S]): Unit but then we wouldn't be able to have a Seq[WriteJobStatsTracker] due to type co-/contra-variance considerations. Instead, you may feel free to just caststatsto the expected derived type when implementing this method in a derived class. The framework will make sure to call this with the right arguments.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()