Package com.wavefront.agent.config
Class LogsIngestionConfig
- java.lang.Object
-
- com.wavefront.agent.config.Configuration
-
- com.wavefront.agent.config.LogsIngestionConfig
-
public class LogsIngestionConfig extends Configuration
Top level configuration object for ingesting log data into the Wavefront Proxy. To turn on logs ingestion, specify 'filebeatPort' and 'logsIngestionConfigFile' in the Wavefront Proxy Config File (typically /etc/wavefront/wavefront-proxy/wavefront.conf, or /opt/wavefront/wavefront-proxy/conf/wavefront.conf). Every file with annotated withJsonProperty
is parsed directly from your logsIngestionConfigFile, which is YAML. Below is a sample config file which shows the features of direct logs ingestion. The "counters" section corresponds tocounters
, likewise forgauges
andhistograms
. In each of these three groups, the pricipal entry is aMetricMatcher
. See the patterns file here for help defining patterns, also various grok debug tools (e.g. this one, or use google) All metrics support dynamic naming with %{}. To see exactly what data we send as part of histograms, seeFlushProcessor.processHistogram(MetricName, Histogram, FlushProcessorContext)
.counters: # For log line "alpha 42", increment counterWithValue by 42 - pattern: "alpha %{NUMBER:value}" metricName: "counterWithValue" # For log line "bravo", increment plainCounter by 1 - pattern: "bravo" metricName: "plainCounter" # For log line "found item in myhost-123", increment "dynamic_myhost_123" by 1 - pattern: "found item in %{WORD:name}-%{NUMBER:type}" metricName: "dynamic_%{name}_%{type}" # For log line "she sold 42 sea shells", increment "shells" by 42. See MYPATTERN below. - pattern: "%{MYPATTERN}" metricName: "shells" valueLabel: "value" # For log line "operation foo took 42 seconds in DC=oregon and AZ=2a", increment "foo.totalSeconds" with point # tags "theDC=oregon theAZ=az-2a" by 42 - pattern: "operation %{WORD:op} took %{NUMBER:value} seconds in DC=%{WORD:dc}.*AZ=%{WORD:az}" metricName: "%{op}.totalSeconds" tagKeys: - "theDC" - "theAZ" tagValues: - "%{dc}" - "az-%{az}" # For log line '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"', # increment apacheBytes by 2326. See the patterns file. - pattern: "%{COMBINEDAPACHELOG}" metricName: "apacheBytes" valueLabel: "bytes" gauges: # For log line "temperature 78", set the "temp" metric to 78 - pattern: "temperature %{NUMBER:value}" metricName: "temp" # For log line: "WARNING: [2878] (SUMMARY): points attempted: 859432; blocked: 0", set wavefrontPointsSent.2878 to 859432 - pattern: '%{LOGLEVEL}: \[%{NUMBER:port}\] %{GREEDYDATA} points attempted: %{NUMBER:pointsAttempted}' metricName: "wavefrontPointsSent.%{port}" valueLabel: "pointsAttempted" histograms: # For log line "histo 123.45", add 123.45 to the running distribution. Will send metrics like p99, mean, ... - pattern: "histo %{NUMBER:value}" metricName: "myHisto" additionalPatterns: - "MYPATTERN she sold %{NUMBER:value} sea shells"
- Author:
- Mori Bellamy ([email protected])
-
-
Field Summary
Fields Modifier and Type Field Description List<String>
additionalPatterns
Additional grok patterns to use in pattern matching for the aboveMetricMatcher
s.Integer
aggregationIntervalSeconds
How often metrics are aggregated and sent to wavefront.int
configReloadIntervalSeconds
How often to check this config file for updates.List<MetricMatcher>
counters
Counters to ingest from incoming log data.long
expiryMillis
Metrics are cleared from memory (and so their aggregation state is lost) if a metric is not updated within this many milliseconds.List<MetricMatcher>
gauges
Gauges to ingest from incoming log data.List<MetricMatcher>
histograms
Histograms to ingest from incoming log data.boolean
reportEmptyHistogramStats
If true (default), simulate Yammer histogram behavior (report all stats as zeroes when histogram is empty).boolean
useDeltaCounters
If true, use delta counters instead of regular counters to prevent metric collisions when multiple proxies are behind a load balancer.boolean
useWavefrontHistograms
If true, useWavefrontHistogram
s rather thanHistogram
s.
-
Constructor Summary
Constructors Constructor Description LogsIngestionConfig()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
verifyAndInit()
-
Methods inherited from class com.wavefront.agent.config.Configuration
ensure, equals, hashCode, toString
-
-
-
-
Field Detail
-
aggregationIntervalSeconds
public Integer aggregationIntervalSeconds
How often metrics are aggregated and sent to wavefront. Histograms are cleared every time they are sent, counters and gauges are not.
-
counters
public List<MetricMatcher> counters
Counters to ingest from incoming log data.
-
gauges
public List<MetricMatcher> gauges
Gauges to ingest from incoming log data.
-
histograms
public List<MetricMatcher> histograms
Histograms to ingest from incoming log data.
-
additionalPatterns
public List<String> additionalPatterns
Additional grok patterns to use in pattern matching for the aboveMetricMatcher
s.
-
expiryMillis
public long expiryMillis
Metrics are cleared from memory (and so their aggregation state is lost) if a metric is not updated within this many milliseconds. Applicable only if useDeltaCounters = false. Default: 3600000 (1 hour).
-
useWavefrontHistograms
public boolean useWavefrontHistograms
If true, useWavefrontHistogram
s rather thanHistogram
s. Histogram ingestion must be enabled on wavefront to use this feature. When using Yammer histograms, the data is exploded into constituent metrics. SeeFlushProcessor.processHistogram(MetricName, Histogram, FlushProcessorContext)
.
-
reportEmptyHistogramStats
public boolean reportEmptyHistogramStats
If true (default), simulate Yammer histogram behavior (report all stats as zeroes when histogram is empty). Otherwise, only .count is reported with a zero value.
-
useDeltaCounters
public boolean useDeltaCounters
If true, use delta counters instead of regular counters to prevent metric collisions when multiple proxies are behind a load balancer. Default: true
-
configReloadIntervalSeconds
public int configReloadIntervalSeconds
How often to check this config file for updates.
-
-
Method Detail
-
verifyAndInit
public void verifyAndInit() throws ConfigurationException
- Specified by:
verifyAndInit
in classConfiguration
- Throws:
ConfigurationException
-
-