Package com.google.cloud.hadoop.fs.gcs

Class GhfsInstrumentation

  • All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.hadoop.fs.statistics.DurationTrackerFactory, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.metrics2.MetricsSource
    public class GhfsInstrumentation
extends Object
implements Closeable, org.apache.hadoop.metrics2.MetricsSource, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.statistics.DurationTrackerFactory
    Instrumentation of GCS.

    Counters and metrics are generally addressed in code by their name or GhfsStatistic key. There may be some Statistics which do not have an entry here. To avoid attempts to access such counters failing, the operations to increment/query metric values are designed to handle lookup failures.

    GoogleHadoopFileSystem StorageStatistics are dynamically derived from the IOStatistics.

    • Constructor Summary

      Constructors 
      Constructor Description
      GhfsInstrumentation​(URI name)
      Construct the instrumentation for a filesystem.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      protected org.apache.hadoop.metrics2.lib.MutableCounterLong counter​(GhfsStatistic op)
      Create a counter in the registry.
      protected org.apache.hadoop.metrics2.lib.MutableCounterLong counter​(String name, String desc)
      Create a counter in the registry.
      void directoryCreated()
      Indicate that GCS created a directory.
      void directoryDeleted()
      Indicate that GCS just deleted a directory.
      protected void duration​(GhfsStatistic op)
      Registering a duration adds the success and failure counters.
      void fileCreated()
      Indicate that GCS created a file.
      void fileDeleted​(int count)
      Indicate that GCS deleted one or more files.
      protected org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge​(GhfsStatistic op)
      Create a gauge in the registry.
      protected org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge​(String name, String desc)
      Create a gauge in the registry.
      org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()
      Get the duration tracker factory.
      org.apache.hadoop.fs.statistics.impl.IOStatisticsStore getIOStatistics()
      Get the instance IO Statistics.
      void getMetrics​(org.apache.hadoop.metrics2.MetricsCollector metricsCollector, boolean b)  
      org.apache.hadoop.metrics2.MetricsSystem getMetricsSystem()
      Get the metrics system.
      org.apache.hadoop.metrics2.lib.MetricsRegistry getRegistry()
      Get the metrics registry.
      void incrementCounter​(GhfsStatistic op, long count)
      Increments a mutable counter and the matching instance IOStatistics counter.
      org.apache.hadoop.metrics2.lib.MutableMetric lookupMetric​(String name)
      Look up a metric from both the registered set and the lighter weight stream entries.
      DelegationTokenStatistics newDelegationTokenStatistics()
      Create a delegation token statistics instance.
      com.google.cloud.hadoop.fs.gcs.GhfsInputStreamStatistics newInputStreamStatistics​(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
      Create a stream input statistics instance.
      com.google.cloud.hadoop.fs.gcs.GhfsOutputStreamStatistics newOutputStreamStatistics​(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
      Create a stream output statistics instance.
      org.apache.hadoop.fs.statistics.DurationTracker trackDuration​(String key, long count)
      The duration tracker updates the metrics with the count and IOStatistics will full duration information.

      • Methods inherited from interface org.apache.hadoop.fs.statistics.DurationTrackerFactory

        trackDuration

    • Constructor Detail

      • GhfsInstrumentation

        public GhfsInstrumentation​(URI name)
        Construct the instrumentation for a filesystem.
        Parameters:
        name - URI of filesystem.

    • Method Detail

      • getIOStatistics

        public org.apache.hadoop.fs.statistics.impl.IOStatisticsStore getIOStatistics()
        Get the instance IO Statistics.
        Specified by:
        getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource
        Returns:
        statistics.

      • incrementCounter

        public void incrementCounter​(GhfsStatistic op,
                             long count)
        Increments a mutable counter and the matching instance IOStatistics counter. No-op if the counter is not defined, or the count == 0.
        Parameters:
        op - operation
        count - increment value

      • getMetricsSystem

        public org.apache.hadoop.metrics2.MetricsSystem getMetricsSystem()
        Get the metrics system.
        Returns:
        metricsSystem

      • counter

        protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter​(String name,
                                                                          String desc)
        Create a counter in the registry.
        Parameters:
        name - counter name
        desc - counter description
        Returns:
        a new counter

      • counter

        protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter​(GhfsStatistic op)
        Create a counter in the registry.
        Parameters:
        op - statistic to count
        Returns:
        a new counter

      • duration

        protected final void duration​(GhfsStatistic op)
        Registering a duration adds the success and failure counters.
        Parameters:
        op - statistic to track

      • gauge

        protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge​(String name,
                                                                      String desc)
        Create a gauge in the registry.
        Parameters:
        name - name gauge name
        desc - description
        Returns:
        the gauge

      • gauge

        protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge​(GhfsStatistic op)
        Create a gauge in the registry.
        Parameters:
        op - statistic to count
        Returns:
        the gauge

      • getRegistry

        public org.apache.hadoop.metrics2.lib.MetricsRegistry getRegistry()
        Get the metrics registry.
        Returns:
        the registry

      • lookupMetric

        public org.apache.hadoop.metrics2.lib.MutableMetric lookupMetric​(String name)
        Look up a metric from both the registered set and the lighter weight stream entries.
        Parameters:
        name - metric name
        Returns:
        the metric or null

      • getDurationTrackerFactory

        public org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()
        Get the duration tracker factory.
        Returns:
        duration tracking for the instrumentation.

      • trackDuration

        public org.apache.hadoop.fs.statistics.DurationTracker trackDuration​(String key,
                                                                     long count)
        The duration tracker updates the metrics with the count and IOStatistics will full duration information.
        Specified by:
        trackDuration in interface org.apache.hadoop.fs.statistics.DurationTrackerFactory
        Parameters:
        key - statistic key prefix
        count - #of times to increment the matching counter in this operation.
        Returns:
        a duration tracker.

      • getMetrics

        public void getMetrics​(org.apache.hadoop.metrics2.MetricsCollector metricsCollector,
                       boolean b)
        Specified by:
        getMetrics in interface org.apache.hadoop.metrics2.MetricsSource

      • fileCreated

        public void fileCreated()
        Indicate that GCS created a file.

      • directoryCreated

        public void directoryCreated()
        Indicate that GCS created a directory.

      • directoryDeleted

        public void directoryDeleted()
        Indicate that GCS just deleted a directory.

      • fileDeleted

        public void fileDeleted​(int count)
        Indicate that GCS deleted one or more files.
        Parameters:
        count - number of files.

      • newInputStreamStatistics

        public com.google.cloud.hadoop.fs.gcs.GhfsInputStreamStatistics newInputStreamStatistics​(@Nullable
                                                                                         org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
        Create a stream input statistics instance.
        Parameters:
        filesystemStatistics - FS Statistics to update in close().
        Returns:
        the new instance

      • newOutputStreamStatistics

        public com.google.cloud.hadoop.fs.gcs.GhfsOutputStreamStatistics newOutputStreamStatistics​(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
        Create a stream output statistics instance.
        Parameters:
        filesystemStatistics - thread-local FS statistics.
        Returns:
        the new instance

      • newDelegationTokenStatistics

        public DelegationTokenStatistics newDelegationTokenStatistics()
        Create a delegation token statistics instance.
        Returns:
        an instance of delegation token statistics