Variables used to store the start and end time of the period of interest for the metrics report
Move data recorded from the custom listener into a DataFrame and register it as a view for easier processing
This inserts the custom Spark Listener into the live Spark Context
Custom aggregations and post-processing of metrics data
for internal metrics sum all the values, for the accumulables compute max value for each accId and name
Custom aggregations and post-processing of metrics data
Shortcut to run and measure the metrics for Spark execution, built after spark.time()
Helper method to save data, we expect to have small amounts of data so collapsing to 1 partition seems OK
Send the metrics to Prometheus.
Send the metrics to Prometheus. serverIPnPort: String with prometheus pushgateway address, format is hostIP:Port, metricsJob: job name, labelName: metrics label name, default is sparkSession.sparkContext.appName, labelValue: metrics label value, default is sparkSession.sparkContext.applicationId