Stage Metrics: collects and aggregates metrics at the end of each stage
Task Metrics: collects data at task granularity
Example usage for task metrics:
val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of
a case class that encapsulates Spark task metrics.
The List Buffer is then transformed into a DataFrame for ease of reporting and analysis.
Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Stage Metrics: collects and aggregates metrics at the end of each stage Task Metrics: collects data at task granularity
Example usage for task metrics: val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark) taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of a case class that encapsulates Spark task metrics. The List Buffer is then transformed into a DataFrame for ease of reporting and analysis.