Type Members

1.  class CgroupMetrics extends SparkPlugin
2.  class CloudFSMetrics extends SparkPlugin

   Monitor S3A I/0 metrics using Hadoop's API FileSystem.getGlobalStorageStatistics FileSystem.getGlobalStorageStatistics has been introduced in Hadoop (client) version 2.8

   Monitor S3A I/0 metrics using Hadoop's API FileSystem.getGlobalStorageStatistics FileSystem.getGlobalStorageStatistics has been introduced in Hadoop (client) version 2.8

   Note: use with Spark 3.x built with Hadoop 3.2 profile (or higher, when available) or use it with Spark built without Hadoop and use your own Hadoop client version 2.8 or higher. Limitation: this currently supports only one filesystem at a time

   Parameters: filesystem name: --conf spark.cernSparkPlugin.cloudFsName=<name of the Hadoop compatible filesystem> (example: s3a) register metrics on the driver conditional to --conf spark.cernSparkPlugin.registerOnDriver=true

3.  class CloudFSMetrics27 extends SparkPlugin

   Instrument Cloud FS I/0 using Hadoop 2.7 client API for FileSystem.getAllStatistics Note this API is deprecated in more recent versions of Hadoop.

   Instrument Cloud FS I/0 using Hadoop 2.7 client API for FileSystem.getAllStatistics Note this API is deprecated in more recent versions of Hadoop. Requires the name of the Hadoop compatible filesystem to measure (s3a, oci, gs, root, etc). Limitation: currently supports only one filesystem at a time

   Parameters: filesystem name: --conf spark.cernSparkPlugin.cloudFsName=<name of the Hadoop compatible filesystem> (example: s3a) register metrics on the driver conditional to --conf spark.cernSparkPlugin.registerOnDriver=true

4.  class DemoMetricsPlugin extends SparkPlugin
5.  class DemoPlugin extends SparkPlugin
6.  class HDFSMetrics extends SparkPlugin

   Monitor HDFS I/0 metrics using Hadoop's API FileSystem.getGlobalStorageStatistics FileSystem.getGlobalStorageStatistics has been introduced in Hadoop (client) version 2.8

   Monitor HDFS I/0 metrics using Hadoop's API FileSystem.getGlobalStorageStatistics FileSystem.getGlobalStorageStatistics has been introduced in Hadoop (client) version 2.8

   Note: use with Spark 3.x built with Hadoop 3.2 profile (or higher, when available) or use it with Spark built without Hadoop and use your own Hadoop client version 2.8 or higher.

7.  class RunOSCommandPlugin extends SparkPlugin