Packages

c

io.github.dutrevis

MemoryMetrics

class MemoryMetrics extends SparkPlugin

Collects memory resource metrics from a unix-based operating system.

Use when Spark is running in clusters with standalone, Mesos or YARN resource managers.

Memory metrics are obtained from the numbers of each line of the /proc/meminfo file, available at the proc pseudo-filesystem of unix-based operating systems. The file has statistics about memory usage on the system, arranged in lines consisted of a parameter name, followed by a colon, the value of the parameter, and an option unit of measurement.

Note

While the /proc/meminfo file shows kilobytes (kB; 1 kB equals 1000 B), its unit is actually kibibytes (KiB; 1 KiB equals 1024 B). This imprecision is known, but is not corrected due to legacy concerns.

,

Many fields have been present since at least Linux 2.6.0, but most of the other fields are available at specific Linux versions (as noted in each method docstring) or are displayed only if the kernel was configured with specific options. If these fields are not found, the metrics won't be registered onto Dropwizard's metric system.

Linear Supertypes
SparkPlugin, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MemoryMetrics
  2. SparkPlugin
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new MemoryMetrics()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def collectBufferMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the Buffers parameter, which is the amount of memory that is used as a relatively temporary storage for raw disk blocks.

    Collects the Buffers parameter, which is the amount of memory that is used as a relatively temporary storage for raw disk blocks.

    Note

    This value shouldn't get tremendously large (20 MB or so).

    ,

    Registered value is in kibibytes and is of type LONG.

  7. def collectCachedSwapMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the SwapCached parameter, which is the amount of swap space used as cache memory - memory that has once been moved into swap, then back into the main memory, but still also remains in the swapfile.

    Collects the SwapCached parameter, which is the amount of swap space used as cache memory - memory that has once been moved into swap, then back into the main memory, but still also remains in the swapfile.

    Note

    Useful memory to save I/O, as it does not need to be moved into swap again.

    ,

    Registered value is in kibibytes and is of type LONG.

  8. def collectFreeMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the MemFree parameter, which is the amount of physical RAM left unused by the system.

    Collects the MemFree parameter, which is the amount of physical RAM left unused by the system.

    Note

    Registered value is in kibibytes and is of type LONG.

  9. def collectFreeSwapMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the SwapFree parameter, which is the amount of swap space currently unused - the memory that has been transferred from RAM to the disk temporarily.

    Collects the SwapFree parameter, which is the amount of swap space currently unused - the memory that has been transferred from RAM to the disk temporarily.

    Note

    Registered value is in kibibytes and is of type LONG.

  10. def collectSharedMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the Shmem parameter - the amount of memory located either for processes with separate address spaces that are sharing a portion of memory, or for tmpfs(5) filesystems, whose contents reside in virtual memory to speed up file access.

    Collects the Shmem parameter - the amount of memory located either for processes with separate address spaces that are sharing a portion of memory, or for tmpfs(5) filesystems, whose contents reside in virtual memory to speed up file access.

    Note

    Available since Linux 2.6.32

    ,

    Registered value is in kibibytes and is of type LONG.

  11. def collectTotalMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the MemTotal parameter, which is the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code).

    Collects the MemTotal parameter, which is the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code).

    Note

    Registered value is in kibibytes and is of type LONG.

  12. def collectTotalSwapMemory(metricCollector: MeminfoMetricCollector): Long

    Collects the SwapTotal parameter, which is the total amount of swap space available on the disk - when the physical memory is full, the system uses the swap space.

    Collects the SwapTotal parameter, which is the total amount of swap space available on the disk - when the physical memory is full, the system uses the swap space.

    Note

    Registered value is in kibibytes and is of type LONG.

  13. def collectUsedMemory(metricCollector: MeminfoMetricCollector): Long

    Calculates the total used memory, by collecting the MemTotal parameter - the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code) - and the MemFree parameter - the amount of physical RAM left unused by the system -, returning the subtraction of the value of the latter out of the first.

    Calculates the total used memory, by collecting the MemTotal parameter - the total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code) - and the MemFree parameter - the amount of physical RAM left unused by the system -, returning the subtraction of the value of the latter out of the first.

    Note

    Registered value is in kibibytes and is of type LONG.

  14. def collectUsedSwapMemory(metricCollector: MeminfoMetricCollector): Long

    Calculates the used swap memory by collecting the SwapTotal parameter - the total amount of swap space available on the disk - and the SwapFree parameter - the total amount of swap space currently unused -, returning the subtraction of the value of the latter out of the first.

    Calculates the used swap memory by collecting the SwapTotal parameter - the total amount of swap space available on the disk - and the SwapFree parameter - the total amount of swap space currently unused -, returning the subtraction of the value of the latter out of the first.

    Note

    Registered value is in kibibytes and is of type LONG.

  15. def createGaugeMetric(metricCollector: MeminfoMetricCollector, collectorMethod: (MeminfoMetricCollector) => Long): Gauge[Long]

    Creates a Dropwizard's Gauge metric type - an instantaneous reading of a particular value -, setting the provided collector method as the getValue method of the Gauge instance.

    Creates a Dropwizard's Gauge metric type - an instantaneous reading of a particular value -, setting the provided collector method as the getValue method of the Gauge instance.

    metricCollector

    a MetricRegistry instance from dropwizard.metrics

    collectorMethod

    a method that receives the metricCollector and returns a metric value as a Long

  16. def driverPlugin(): DriverPlugin

    Returns the plugin's driver-side component.

    Returns the plugin's driver-side component. The returned DriverPlugin instance is called once, early in the initialization of the Spark driver. The operation it performs consists in the sequential registration of each mapped metric as a Gauge metric into an existing metric Registry. A test call is executed once on each collector method before its registration, assuring that the metric is available to be read and collected from the local OS, thus preventing future errors when the Gauge is first executed by the metrics system. If a NoSuchElementException is thrown in this attempt, the method is not registered, enabling the registration of the subsequent mapped metrics by the plugin. The plugin component ends its execution once all metrics are registered, leaving to the Dropwizard's Metrics system the job of collecting and exporting the registered metrics in a pre or user-defined frequency.

    returns

    An instance of the DriverPlugin

    Definition Classes
    MemoryMetrics → SparkPlugin
    Note

    The driver's initialization is blocked during the operations inside init, so heavy performing operations must be avoided.

    ,

    The overriden init method must return a Map, that will be provided as extraConf to an ExecutorPlugin instance.

  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  19. def executorPlugin(): ExecutorPlugin

    Returns the plugin's executor-side component.

    Returns the plugin's executor-side component. The returned ExecutorPlugin instance is called once, early in the initialization of the executor process. The operation it performs consists in the sequential registration of each mapped metric as a Gauge metric into an existing metric Registry. A test call is executed once on each collector method before its registration, assuring that the metric is available to be read and collected from the local OS, thus preventing future errors when the Gauge is first executed by the metrics system. If a NoSuchElementException is thrown in this attempt, the method is not registered, enabling the registration of the subsequent mapped metrics by the plugin. The plugin component ends its execution once all metrics are registered, leaving to the Dropwizard's Metrics system the job of collecting and exporting the registered metrics in a pre or user-defined frequency.

    returns

    An instance of the ExecutorPlugin Unit

    Definition Classes
    MemoryMetrics → SparkPlugin
    Note

    The executor's initialization is blocked during the operations inside init, so heavy performing operations must be avoided.

  20. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  21. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. val metricMapping: Map[String, (MeminfoMetricCollector) => Long]

    Maps the collector methods to their respective metric names, that will be displayed in the Dropwizard's metric system.

  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  26. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  27. def registerMetric(metricRegistry: MetricRegistry, metricName: String, metricInstance: Metric): Unit

    Registers a provided Dropwizard's Metric instance into a metric registry under a metric name.

    Registers a provided Dropwizard's Metric instance into a metric registry under a metric name.

    metricRegistry

    a MetricRegistry instance from dropwizard.metrics

    metricName

    a metric name as a String

    metricInstance

    an instance of a dropwizard's Metric class to be registered

    Exceptions thrown

    IllegalArgumentException if the metric name is already registered

  28. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  29. def toString(): String
    Definition Classes
    AnyRef → Any
  30. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  31. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  32. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from SparkPlugin

Inherited from AnyRef

Inherited from Any

Ungrouped