TaskMetrics
Related Doc:
package executor
The number of on-disk bytes spilled by this task.
CPU Time the executor spends actually running the task (including fetching shuffle data) in nanoseconds.
CPU Time taken on the executor to deserialize this task in nanoseconds.
Time taken on the executor to deserialize this task.
Time the executor spends actually running the task (including fetching shuffle data).
Metrics related to reading data from a org.apache.spark.rdd.HadoopRDD or from persisted data, defined only in tasks with input.
Amount of time the JVM spent in garbage collection while executing this task.
The number of in-memory bytes spilled by this task.
Metrics related to writing data externally (e.g.
Metrics related to writing data externally (e.g. to a distributed filesystem), defined only in tasks with output.
Peak memory used by internal data structures created during shuffles, aggregations and joins.
Peak memory used by internal data structures created during shuffles, aggregations and joins. The value of this accumulator should be approximately the sum of the peak sizes across all such data structures created in this task. For SQL jobs, this only tracks all unsafe operators and ExternalSort.
Amount of time spent serializing the task result.
The number of bytes this task transmitted back to the driver as the TaskResult.
Metrics related to shuffle read aggregated across all shuffle dependencies.
Metrics related to shuffle read aggregated across all shuffle dependencies. This is defined only if there are shuffle dependencies in this task.
Metrics related to shuffle write, defined only in shuffle map stages.
Storage statuses of any blocks that have been updated as a result of this task.
:: DeveloperApi :: Metrics tracked during the execution of a task.
This class is wrapper around a collection of internal accumulators that represent metrics associated with a task. The local values of these accumulators are sent from the executor to the driver when the task completes. These values are then merged into the corresponding accumulator previously registered on the driver.
The accumulator updates are also sent to the driver periodically (on executor heartbeat) and when the task failed with an exception. The TaskMetrics object itself should never be sent to the driver.