Package com.wavefront.agent.config

Class LogsIngestionConfig

  • public class LogsIngestionConfig
extends Configuration
    Top level configuration object for ingesting log data into the Wavefront Proxy. To turn on logs ingestion, specify 'filebeatPort' and 'logsIngestionConfigFile' in the Wavefront Proxy Config File (typically /etc/wavefront/wavefront-proxy/wavefront.conf, or /opt/wavefront/wavefront-proxy/conf/wavefront.conf). Every file with annotated with JsonProperty is parsed directly from your logsIngestionConfigFile, which is YAML. Below is a sample config file which shows the features of direct logs ingestion. The "counters" section corresponds to counters, likewise for gauges and histograms. In each of these three groups, the pricipal entry is a MetricMatcher. See the patterns file here for help defining patterns, also various grok debug tools (e.g. this one, or use google) All metrics support dynamic naming with %{}. To see exactly what data we send as part of histograms, see FlushProcessor.processHistogram(MetricName, Histogram, FlushProcessorContext). 
     
 counters:
   # For log line "alpha 42", increment counterWithValue by 42
   - pattern: "alpha %{NUMBER:value}"
     metricName: "counterWithValue"
   # For log line "bravo", increment plainCounter by 1
   - pattern: "bravo"
     metricName: "plainCounter"
   # For log line "found item in myhost-123", increment "dynamic_myhost_123" by 1
   - pattern: "found item in %{WORD:name}-%{NUMBER:type}"
     metricName: "dynamic_%{name}_%{type}"
   # For log line "she sold 42 sea shells", increment "shells" by 42. See MYPATTERN below.
   - pattern: "%{MYPATTERN}"
     metricName: "shells"
     valueLabel: "value"
   # For log line "operation foo took 42 seconds in DC=oregon and AZ=2a", increment "foo.totalSeconds" with point
   # tags "theDC=oregon theAZ=az-2a" by 42
   - pattern: "operation %{WORD:op} took %{NUMBER:value} seconds in DC=%{WORD:dc}.*AZ=%{WORD:az}"
     metricName: "%{op}.totalSeconds"
     tagKeys:
       - "theDC"
       - "theAZ"
     tagValues:
       - "%{dc}"
       - "az-%{az}"
   # For log line '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"',
   # increment apacheBytes by 2326. See the patterns file.
   - pattern: "%{COMBINEDAPACHELOG}"
     metricName: "apacheBytes"
     valueLabel: "bytes"

 gauges:
   # For log line "temperature 78", set the "temp" metric to 78
   - pattern: "temperature %{NUMBER:value}"
     metricName: "temp"
   # For log line: "WARNING: [2878] (SUMMARY): points attempted: 859432; blocked: 0", set wavefrontPointsSent.2878 to 859432
   - pattern: '%{LOGLEVEL}: \[%{NUMBER:port}\] %{GREEDYDATA} points attempted: %{NUMBER:pointsAttempted}'
     metricName: "wavefrontPointsSent.%{port}"
     valueLabel: "pointsAttempted"

 histograms:
   # For log line "histo 123.45", add 123.45 to the running distribution. Will send metrics like p99, mean, ...
   - pattern: "histo %{NUMBER:value}"
     metricName: "myHisto"

 additionalPatterns:
   - "MYPATTERN she sold %{NUMBER:value} sea shells"
    Author:
    Mori Bellamy ([email protected])

    • Field Detail

      • aggregationIntervalSeconds

        public Integer aggregationIntervalSeconds
        How often metrics are aggregated and sent to wavefront. Histograms are cleared every time they are sent, counters and gauges are not.

      • counters

        public List<MetricMatcher> counters
        Counters to ingest from incoming log data.

      • histograms

        public List<MetricMatcher> histograms
        Histograms to ingest from incoming log data.

      • additionalPatterns

        public List<String> additionalPatterns
        Additional grok patterns to use in pattern matching for the above MetricMatchers.

      • expiryMillis

        public long expiryMillis
        Metrics are cleared from memory (and so their aggregation state is lost) if a metric is not updated within this many milliseconds. Applicable only if useDeltaCounters = false. Default: 3600000 (1 hour).

      • reportEmptyHistogramStats

        public boolean reportEmptyHistogramStats
        If true (default), simulate Yammer histogram behavior (report all stats as zeroes when histogram is empty). Otherwise, only .count is reported with a zero value.

      • useDeltaCounters

        public boolean useDeltaCounters
        If true, use delta counters instead of regular counters to prevent metric collisions when multiple proxies are behind a load balancer. Default: true

      • configReloadIntervalSeconds

        public int configReloadIntervalSeconds
        How often to check this config file for updates.

    • Constructor Detail

      • LogsIngestionConfig

        public LogsIngestionConfig()