class MemoryMetrics extends SparkPlugin
Collects memory resource metrics from a unix-based operating system.
Use when Spark is running in clusters with standalone, Mesos or YARN resource managers.
Memory metrics are obtained from the numbers of each line of
the /proc/meminfo
file, available at the proc pseudo-filesystem of
unix-based operating systems. The file has statistics about memory usage on
the system, arranged in lines consisted of a parameter name, followed by a
colon, the value of the parameter, and an option unit of measurement.
- Note
While the
,/proc/meminfo
file shows kilobytes (kB; 1 kB equals 1000 B), its unit is actually kibibytes (KiB; 1 KiB equals 1024 B). This imprecision is known, but is not corrected due to legacy concerns.Many fields have been present since at least Linux 2.6.0, but most of the other fields are available at specific Linux versions (as noted in each method docstring) or are displayed only if the kernel was configured with specific options. If these fields are not found, the metrics won't be registered onto Dropwizard's metric system.
- Alphabetic
- By Inheritance
- MemoryMetrics
- SparkPlugin
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new MemoryMetrics()
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def collectBufferMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
Buffers
parameter, which is the amount of memory that is used as a relatively temporary storage for raw disk blocks.Collects the
Buffers
parameter, which is the amount of memory that is used as a relatively temporary storage for raw disk blocks.- Note
This value shouldn't get tremendously large (20 MB or so).
,Registered value is in kibibytes and is of type LONG.
- def collectCachedSwapMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
SwapCached
parameter, which is the amount of swap space used as cache memory - memory that has once been moved into swap, then back into the main memory, but still also remains in the swapfile.Collects the
SwapCached
parameter, which is the amount of swap space used as cache memory - memory that has once been moved into swap, then back into the main memory, but still also remains in the swapfile.- Note
Useful memory to save I/O, as it does not need to be moved into swap again.
,Registered value is in kibibytes and is of type LONG.
- def collectFreeMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
MemFree
parameter, which is the amount of physical RAM left unused by the system.Collects the
MemFree
parameter, which is the amount of physical RAM left unused by the system.- Note
Registered value is in kibibytes and is of type LONG.
- def collectFreeSwapMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
SwapFree
parameter, which is the amount of swap space currently unused - the memory that has been transferred from RAM to the disk temporarily.Collects the
SwapFree
parameter, which is the amount of swap space currently unused - the memory that has been transferred from RAM to the disk temporarily.- Note
Registered value is in kibibytes and is of type LONG.
- def collectSharedMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
Shmem
parameter - the amount of memory located either for processes with separate address spaces that are sharing a portion of memory, or for tmpfs(5) filesystems, whose contents reside in virtual memory to speed up file access.Collects the
Shmem
parameter - the amount of memory located either for processes with separate address spaces that are sharing a portion of memory, or for tmpfs(5) filesystems, whose contents reside in virtual memory to speed up file access.- Note
Available since Linux 2.6.32
,Registered value is in kibibytes and is of type LONG.
- def collectTotalMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
MemTotal
parameter, which is the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code).Collects the
MemTotal
parameter, which is the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code).- Note
Registered value is in kibibytes and is of type LONG.
- def collectTotalSwapMemory(metricCollector: MeminfoMetricCollector): Long
Collects the
SwapTotal
parameter, which is the total amount of swap space available on the disk - when the physical memory is full, the system uses the swap space.Collects the
SwapTotal
parameter, which is the total amount of swap space available on the disk - when the physical memory is full, the system uses the swap space.- Note
Registered value is in kibibytes and is of type LONG.
- def collectUsedMemory(metricCollector: MeminfoMetricCollector): Long
Calculates the total used memory, by collecting the
MemTotal
parameter - the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code) - and theMemFree
parameter - the amount of physical RAM left unused by the system -, returning the subtraction of the value of the latter out of the first.Calculates the total used memory, by collecting the
MemTotal
parameter - the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code) - and theMemFree
parameter - the amount of physical RAM left unused by the system -, returning the subtraction of the value of the latter out of the first.- Note
Registered value is in kibibytes and is of type LONG.
- def collectUsedSwapMemory(metricCollector: MeminfoMetricCollector): Long
Calculates the used swap memory by collecting the
SwapTotal
parameter - the total amount of swap space available on the disk - and theSwapFree
parameter - the total amount of swap space currently unused -, returning the subtraction of the value of the latter out of the first.Calculates the used swap memory by collecting the
SwapTotal
parameter - the total amount of swap space available on the disk - and theSwapFree
parameter - the total amount of swap space currently unused -, returning the subtraction of the value of the latter out of the first.- Note
Registered value is in kibibytes and is of type LONG.
- def createGaugeMetric(metricCollector: MeminfoMetricCollector, collectorMethod: (MeminfoMetricCollector) => Long): Gauge[Long]
Creates a Dropwizard's Gauge metric type - an instantaneous reading of a particular value -, setting the provided collector method as the
getValue
method of the Gauge instance.Creates a Dropwizard's Gauge metric type - an instantaneous reading of a particular value -, setting the provided collector method as the
getValue
method of the Gauge instance.- metricCollector
a MetricRegistry instance from dropwizard.metrics
- collectorMethod
a method that receives the metricCollector and returns a metric value as a Long
- def driverPlugin(): DriverPlugin
Returns the plugin's driver-side component.
Returns the plugin's driver-side component. The returned DriverPlugin instance is called once, early in the initialization of the Spark driver. The operation it performs consists in the sequential registration of each mapped metric as a Gauge metric into an existing metric Registry. A test call is executed once on each collector method before its registration, assuring that the metric is available to be read and collected from the local OS, thus preventing future errors when the Gauge is first executed by the metrics system. If a
NoSuchElementException
is thrown in this attempt, the method is not registered, enabling the registration of the subsequent mapped metrics by the plugin. The plugin component ends its execution once all metrics are registered, leaving to the Dropwizard's Metrics system the job of collecting and exporting the registered metrics in a pre or user-defined frequency.- returns
An instance of the
DriverPlugin
- Definition Classes
- MemoryMetrics → SparkPlugin
- Note
The driver's initialization is blocked during the operations inside
,init
, so heavy performing operations must be avoided.The overriden
init
method must return a Map, that will be provided asextraConf
to anExecutorPlugin
instance.
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def executorPlugin(): ExecutorPlugin
Returns the plugin's executor-side component.
Returns the plugin's executor-side component. The returned ExecutorPlugin instance is called once, early in the initialization of the executor process. The operation it performs consists in the sequential registration of each mapped metric as a Gauge metric into an existing metric Registry. A test call is executed once on each collector method before its registration, assuring that the metric is available to be read and collected from the local OS, thus preventing future errors when the Gauge is first executed by the metrics system. If a
NoSuchElementException
is thrown in this attempt, the method is not registered, enabling the registration of the subsequent mapped metrics by the plugin. The plugin component ends its execution once all metrics are registered, leaving to the Dropwizard's Metrics system the job of collecting and exporting the registered metrics in a pre or user-defined frequency.- returns
An instance of the
ExecutorPlugin
Unit
- Definition Classes
- MemoryMetrics → SparkPlugin
- Note
The executor's initialization is blocked during the operations inside
init
, so heavy performing operations must be avoided.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val metricMapping: Map[String, (MeminfoMetricCollector) => Long]
Maps the collector methods to their respective metric names, that will be displayed in the Dropwizard's metric system.
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def registerMetric(metricRegistry: MetricRegistry, metricName: String, metricInstance: Metric): Unit
Registers a provided Dropwizard's Metric instance into a metric registry under a metric name.
Registers a provided Dropwizard's Metric instance into a metric registry under a metric name.
- metricRegistry
a MetricRegistry instance from dropwizard.metrics
- metricName
a metric name as a String
- metricInstance
an instance of a dropwizard's Metric class to be registered
- Exceptions thrown
IllegalArgumentException
if the metric name is already registered
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)