com.wavefront.agent.config

Class LogsIngestionConfig

java.lang.Object

com.wavefront.agent.config.Configuration

com.wavefront.agent.config.LogsIngestionConfig

public class LogsIngestionConfig
extends Configuration

Top level configuration object for ingesting log data into the Wavefront Proxy. To turn on logs ingestion, specify 'filebeatPort' and 'logsIngestionConfigFile' in the Wavefront Proxy Config File (typically /etc/wavefront/wavefront-proxy/wavefront.conf, or /opt/wavefront/wavefront-proxy/conf/wavefront.conf). Every file with annotated with JsonProperty is parsed directly from your logsIngestionConfigFile, which is YAML. Below is a sample config file which shows the features of direct logs ingestion. The "counters" section corresponds to counters, likewise for gauges and histograms. In each of these three groups, the pricipal entry is a MetricMatcher. See the patterns file here for help defining patterns, also various grok debug tools (e.g. this one, or use google) All metrics support dynamic naming with %{}. To see exactly what data we send as part of histograms, see FlushProcessor.processHistogram(MetricName, Histogram, FlushProcessorContext).

 
 counters:
   # For log line "alpha 42", increment counterWithValue by 42
   - pattern: "alpha %{NUMBER:value}"
     metricName: "counterWithValue"
   # For log line "bravo", increment plainCounter by 1
   - pattern: "bravo"
     metricName: "plainCounter"
   # For log line "found item in myhost-123", increment "dynamic_myhost_123" by 1
   - pattern: "found item in %{WORD:name}-%{NUMBER:type}"
     metricName: "dynamic_%{name}_%{type}"
   # For log line "she sold 42 sea shells", increment "shells" by 42. See MYPATTERN below.
   - pattern: "%{MYPATTERN}"
     metricName: "shells"
     valueLabel: "value"
   # For log line "operation foo took 42 seconds in DC=oregon and AZ=2a", increment "foo.totalSeconds" with point
   # tags "theDC=oregon theAZ=az-2a" by 42
   - pattern: "operation %{WORD:op} took %{NUMBER:value} seconds in DC=%{WORD:dc}.*AZ=%{WORD:az}"
     metricName: "%{op}.totalSeconds"
     tagKeys:
       - "theDC"
       - "theAZ"
     tagValues:
       - "%{dc}"
       - "az-%{az}"
   # For log line '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"',
   # increment apacheBytes by 2326. See the patterns file.
   - pattern: "%{COMBINEDAPACHELOG}"
     metricName: "apacheBytes"
     valueLabel: "bytes"

 gauges:
   # For log line "temperature 78", set the "temp" metric to 78
   - pattern: "temperature %{NUMBER:value}"
     metricName: "temp"
   # For log line: "WARNING: [2878] (SUMMARY): points attempted: 859432; blocked: 0", set wavefrontPointsSent.2878 to 859432
   - pattern: '%{LOGLEVEL}: \[%{NUMBER:port}\] %{GREEDYDATA} points attempted: %{NUMBER:pointsAttempted}'
     metricName: "wavefrontPointsSent.%{port}"
     valueLabel: "pointsAttempted"

 histograms:
   # For log line "histo 123.45", add 123.45 to the running distribution. Will send metrics like p99, mean, ...
   - pattern: "histo %{NUMBER:value}"
     metricName: "myHisto"

 additionalPatterns:
   - "MYPATTERN she sold %{NUMBER:value} sea shells"
 
 

Author:

Mori Bellamy ([email protected])

Field Summary

Fields

Modifier and Type	Field and Description
List<String>	additionalPatterns Additional grok patterns to use in pattern matching for the above MetricMatchers.
Integer	aggregationIntervalSeconds How often metrics are aggregated and sent to wavefront.
int	configReloadIntervalSeconds How often to check this config file for updates.
List<MetricMatcher>	counters Counters to ingest from incoming log data.
long	expiryMillis Metrics are cleared from memory (and so their aggregation state is lost) if a metric is not updated within this many milliseconds.
List<MetricMatcher>	gauges Gauges to ingest from incoming log data.
List<MetricMatcher>	histograms Histograms to ingest from incoming log data.
boolean	reportEmptyHistogramStats If true (default), simulate Yammer histogram behavior (report all stats as zeroes when histogram is empty).
boolean	useDeltaCounters If true, use delta counters instead of regular counters to prevent metric collisions when multiple proxies are behind a load balancer.
boolean	useWavefrontHistograms If true, use WavefrontHistograms rather than Histograms.

Constructor Summary

Constructors

Constructor and Description
LogsIngestionConfig()

Method Summary

Modifier and Type	Method and Description
void	verifyAndInit()

Methods inherited from class com.wavefront.agent.config.Configuration

ensure, equals, hashCode, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

aggregationIntervalSeconds

public Integer aggregationIntervalSeconds

How often metrics are aggregated and sent to wavefront. Histograms are cleared every time they are sent, counters and gauges are not.

counters

public List<MetricMatcher> counters

Counters to ingest from incoming log data.

gauges

public List<MetricMatcher> gauges

Gauges to ingest from incoming log data.

histograms

public List<MetricMatcher> histograms

Histograms to ingest from incoming log data.

additionalPatterns

public List<String> additionalPatterns

Additional grok patterns to use in pattern matching for the above MetricMatchers.

expiryMillis

public long expiryMillis

Metrics are cleared from memory (and so their aggregation state is lost) if a metric is not updated within this many milliseconds. Applicable only if useDeltaCounters = false. Default: 3600000 (1 hour).

useWavefrontHistograms

public boolean useWavefrontHistograms

If true, use WavefrontHistograms rather than Histograms. Histogram ingestion must be enabled on wavefront to use this feature. When using Yammer histograms, the data is exploded into constituent metrics. See FlushProcessor.processHistogram(MetricName, Histogram, FlushProcessorContext).

reportEmptyHistogramStats

public boolean reportEmptyHistogramStats

If true (default), simulate Yammer histogram behavior (report all stats as zeroes when histogram is empty). Otherwise, only .count is reported with a zero value.

useDeltaCounters

public boolean useDeltaCounters

If true, use delta counters instead of regular counters to prevent metric collisions when multiple proxies are behind a load balancer. Default: true

configReloadIntervalSeconds

public int configReloadIntervalSeconds

How often to check this config file for updates.

Constructor Detail

LogsIngestionConfig

public LogsIngestionConfig()

Method Detail

verifyAndInit

public void verifyAndInit()
                   throws ConfigurationException

Specified by:

verifyAndInit in class Configuration

Throws:

ConfigurationException