telegraf #39
Description
Telegraf is an agent written in Go for collecting metrics from the system it's running on, or from other services, and writing them into InfluxDB or other outputs. Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics from well known services (like Hadoop, Postgres, or Redis) and third party APIs (like Mailchimp, AWS CloudWatch, or Google Analytics). New input and output plugins are designed to be easy to contribute, we'll eagerly accept pull requests and will manage the set of plugins that Telegraf supports. See the contributing guide for instructions on writing new plugins.
- Tags:
- monitoring ›
Overview
This is a subordinate charm to deploy the telegraf metrics agent to collect metrics from related services. The metrics may be scraped by or pushed to supported systems such as Prometheus, Graphite or InfluxDB.
For details about telegraf see: https://github.com/influxdata/telegraf
Usage
Deploy telegraf alongside your service, and also a time series storage (in this case, influxdb)
juju deploy telegraf
juju deploy influxdb
juju deploy some-service
Add the relations:
juju add-relation telegraf:juju-info some-service:juju-info
juju add-relation telegraf:influxdb-api influxdb:api
Network space support
The charm support the 'prometheus-client' binding for the prometheus-client interface. If the charm is deployed with 'prometheus-client' bound to a network space then the client relation will contain the network space IP address rather than the unit's private address.
Configuration
Installation method: deb or snap
The charm can deploy using either a deb (default) or a snap installation. juju config telegraf install_method=deb # to use the debian package juju config telegraf install_method=snap # to use the snap Only the needed package (deb or snap) will be installed.
To change the snap channel: juju config telegraf snap_channel=beta
Prometheus Output
By default, Prometheus output is presented available for scraping.
InfluxDB Output
When related to an InfluxDB service, metrics will be pushed to a database
named telegraf
.
Apache Input
For the apache input plugin, the charm provides the apache relation which
uses apache-website
interface. Current apache charm disables
mod_status
and in order to telegraf apache input to work status
should be removed from the list of disable_modules
in the apache charm
config.
Postgresql & MySQL input
Due to relation scope issues,
two relations need to be established between telegraf and the postgresql
service. first a plain juju-info
relation to get standard system
metrics and a db-admin
relation for PostgreSQL database metrics. eg.:
```
juju add-relation postgresql:juju-info telegraf
juju add-relation postgresql:db-admin telegraf
```
Similarly, for MySQL:
```
juju add-relation mysql:juju-info telegraf
juju add-relation mysql:db-admin telegraf
```
Rabbitmq Input
You will need both relations:
* juju add-relation telegraf:juju-info rabbitmq-server:juju-info
to start serving data
* juju add-relation telegraf:amqp rabbitmq-server:amqp
for the actual rabbitmq communication
This relation requires rabbitmq management plugin to be activated: juju set rabbitmq-server management_plugin=true
The rabbitmq charm doesn't appear to give true administrator privileges to relations.
On the rabbitmq unit, you will need to grant permissions for the newly created charm user: rabbitmqctl set_permissions -p openstack telegraf-telegraf-XX ".*" ".*" ".*"
The charm configures telegraf to drop idle_since
metrics (e.g. idle_since="2018-08-14 0:24:04"
) which break time series in Prometheus, since any difference in labels (presence or lack, and content) is treated as a new series.
Advanced
Input
A basic set of input plugins are setup by default, which can be overriden
with the inputs_config
configuration option. Both of these accept raw
telegraf configuration.
To configure any of the (default or via relation) plugins, the
extra_options
charm config can be used. It's yaml, that needs to
be encoded as a string in the config. For example:
```yaml
telegraf:
# The trailing | means the following indented section is all one string.
extra_options: |
inputs:
cpu:
percpu: false
fielddrop: ["time_*"]
disk:
mount_points: ["/"]
ignore_fs: ["tmpfs", "devtmpfs"]
elasticsearch:
local: false
cluster_health: true
postgresql:
databases: ["foo", "bar"]
tagpass:
db: ["template", "postgres"]
outputs:
influxdb:
precision: ms
```
This extra options will only be applied to plugins configured via
relations and the following list (defined in templates/base_inputs.conf
):
- inputs.cpu
- inputs.disk
- inputs.diskio
- inputs.mem
- inputs.net
- inputs.netstat
- inputs.swap
- inputs.system
Output
The only output plugin supported via relation is influxdb, any other output plugin needs to be configured manually (via juju set).
To use a different metrics storage such as Graphite, the raw telegraf
plugin configuration needs to be set as a string in outputs_config
configuration. For example:
```yaml
telegraf:
# The trailing | indicates a multiline string in yaml.
outputs_config: |
[[outputs.graphite]]
servers = ["10.0.3.231:2003"]
prefix = "juju_local.devel.telegraf"
timeout = 10
```
This will make telegraf agents to send the metrics to the graphite instance.
Contact Information
- Prometheus Charming Project: https://launchpad.net/prometheus-charms
- Charm bugs: https://bugs.launchpad.net/telegraf-charm
- Upstream https://github.com/influxdata/telegraf
- Upstream bug tracker https://github.com/influxdata/telegraf/issues
Configuration
- collection_jitter
- (string) Collection jitter is used to jitter the collection by a random amount. Each plugin will sleep for a random time within jitter before collecting. This can be used to avoid many plugins querying things like sysfs at the same time, which can have a measurable effect on the system.
- 0s
- cpu_idle
- (int) Percent of idle CPU at which to generate a Prometheus alert. Default 10%, i.e. alert if the CPU has less than 10% CPU running idle therefore 90% used by us+sy+wa+st.
- 10
- debug
- (boolean) Run telegraf in debug mode
- disabled_plugins
- (string) Colon(:) separated plugins names to disable, including builtin input plugins and custom exec metrics in `files/telegraf_exec_metrics.py`, e.g.: cpu:mem:buddyinfo:zoneinfo
- extra_options
- (string) YAML with extra options for out|inputs managed by relations or in the default config. example: inputs: cpu: percpu: false fielddrop: ["time_*"] disk: mount_points: ["/"] ignore_fs: ["tmpfs", "devtmpfs"] elasticsearch: local: false cluster_health: true postgresql: databases: ["foo", "bar"] tagpass: db: ["template", "postgres"] outputs: influxdb: precision: ms
- extra_packages
- (string) Space separated list of extra deb packages to install.
- extra_plugins
- (string) Extra plugins, manually configured. This is expected to be a string and will be saved "as is" in telegraf.d/extra_plugins.conf
- flush_interval
- (string) Default data flushing interval for all outputs. You should not set this below interval. Maximum flush_interval will be flush_interval + flush_jitter
- 10s
- flush_jitter
- (string) Jitter the flush interval by a random amount. This is primarily to avoid large write spikes for users running a large number of telegraf instances. ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
- 0s
- hostname
- (string) Override default hostname. {model} is replaced with the Juju model name and {unit} is replaced with the sanitized unit name (eg. service_name-0). {uuid} is replaced by the model UUID, for sites without unique model names. {host} is replaced by the machine hostname
- {model}:{unit}
- inputs_config
- (string) [inputs.xxx] sections as a string, this override default input plugins.
- install_keys
- (string) List of signing keys for install_sources package sources, per charmhelpers standard format (a yaml list of strings encoded as a string). The keys should be the full ASCII armoured GPG public keys. While GPG key ids are also supported and looked up on a keyserver, operators should be aware that this mechanism is insecure. null can be used if a standard package signing key is used that will already be installed on the machine, and for PPA sources where the package signing key is securely retrieved from Launchpad.
- - null
- install_method
- (string) Installation method. Valid values are "deb" or "snap".
- deb
- install_sources
- (string) List of extra apt sources, per charm-helpers standard format (a yaml list of strings encoded as a string). Each source may be either a line that can be added directly to sources.list(5), or in the form ppa:<user>/<ppa-name> for adding Personal Package Archives, or a distribution component to enable.
- - ppa:telegraf-devs/ppa
- interval
- (string) Default data collection interval for all plugins
- 10s
- lead_time
- (string) Time period to use when alerting on predictive Prometheus rules, e.g a disk might run low on space in $lead_time, based on the previous usage over lead_time. Default 14d, i.e. based on the previous 14 days, alert if we have 14 days disk left. String, e.g. 14d, 4w.
- 14d
- logfile
- (string) Name of the file to be logged to when using the file logtarget. If set to the empty string then logs are written to stderr.
- /var/log/telegraf/telegraf.log
- metric_buffer_limit
- (int) Telegraf will cache metric_buffer_limit metrics for each output, and will flush this buffer on a successful write.
- 10000
- nagios_context
- (string) A string that will be prepended to instance name to set the host name in nagios. So for instance the hostname would be something like: juju-myservice-0 If you're running multiple environments with the same services in them this allows you to differentiate between them.
- juju
- nagios_servicegroups
- (string) Comma separated list of nagios servicegroups for the telegraf check
- juju
- outputs_config
- (string) [outputs.xxx] sections as a string
- package_status
- (string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".
- install
- prometheus_context
- (string) A string that will be used to identify this site in the 'cloud_name' label in Prometheus. If you're running multiple environments with the same services in them this allows you to differentiate between them.
- juju
- prometheus_datasource
- (string) Specifies the datasource for Grafana dashboards. This usually corresponds to the application name of the related Prometheus. Defaults to "prometheus". Note: The complete datasource inserted into dashboard will be "$prometheus_datasource - Juju generated source" This matches how the Prometheus2/Grafana charms format the datasource.
- prometheus
- prometheus_output_port
- (string) If set prometheus output plugin will be configured to listen on the provided port. If set to string "default" the charm will use default port (9103)
- 9103
- promreg_authtoken
- (string) The Prometheus Registration authtoken.
- promreg_url
- (string) The http(s) url and port where the Prometheus Registration service is listening.
- quiet
- (boolean) Run telegraf in quiet mode
- round_interval
- (boolean) Rounds collection interval to 'interval' ie, if interval="10s" then always collect on :00, :10, :20, etc.
- True
- snap_channel
- (string) Snap Store channel from which to install the telegraf snap.
- stable
- snap_proxy
- (string) DEPRECATED. Use snap-http-proxy and snap-https-proxy model configuration settings. HTTP/HTTPS web proxy for Snappy to use when accessing the snap store.
- snap_proxy_url
- (string) DEPRECATED. Use snap-store-proxy model configuration setting. The address of a Snap Store Proxy to use for snaps e.g. http://snap-proxy.example.com
- snapd_refresh
- (string) How often snapd handles updates for installed snaps. The default (an empty string) is 4x per day. Set to "max" to check once per month based on the charm deployment date. You may also set a custom string as described in the 'refresh.timer' section here: https://forum.snapcraft.io/t/system-options/87
- socket_listener_port
- (string) If set, the input socket_listener will listen on the specified port. The default option is to listen to 8094.
- 8094
- (string) Comma separated list of global tags. ie, 'dc=us-east-1,rack=1a' will tag all metrics with dc=us-east-1 and rack=1a
- wait_time
- (string) When exporting Prometheus alerting rules, how long a condition should be true before firing an alert. String, e.g. 5m, 1h, etc.
- 5m