Stage Metrics: collects and aggregates metrics at the end of each stage
Task Metrics: collects data at task granularity
Example usage for stage metrics:
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of
a case class that encapsulates Spark task metrics.
The List Buffer is then transformed into a DataFrame for ease of reporting and analysis.
Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Stage Metrics: collects and aggregates metrics at the end of each stage Task Metrics: collects data at task granularity
Example usage for stage metrics: val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of a case class that encapsulates Spark task metrics. The List Buffer is then transformed into a DataFrame for ease of reporting and analysis.