telegraf #41

Supports: focal bionic xenial

Description

Telegraf is an agent written in Go for collecting metrics from the system it's running on, or from other services, and writing them into InfluxDB or other outputs. Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics from well known services (like Hadoop, Postgres, or Redis) and third party APIs (like Mailchimp, AWS CloudWatch, or Google Analytics). New input and output plugins are designed to be easy to contribute, we'll eagerly accept pull requests and will manage the set of plugins that Telegraf supports. See the contributing guide for instructions on writing new plugins.

Overview

This is a subordinate charm to deploy the telegraf metrics agent to collect metrics from related services. The metrics may be scraped by or pushed to supported systems such as Prometheus, Graphite or InfluxDB.

For details about telegraf see: https://github.com/influxdata/telegraf

Usage

Deploy telegraf alongside your service, and also a time series storage (in this case, influxdb)

juju deploy telegraf
juju deploy influxdb
juju deploy some-service

Add the relations:

juju add-relation telegraf:juju-info some-service:juju-info
juju add-relation telegraf:influxdb-api influxdb:api

Network space support

The charm support the 'prometheus-client' binding for the prometheus-client interface. If the charm is deployed with 'prometheus-client' bound to a network space then the client relation will contain the network space IP address rather than the unit's private address.

Note for deployment into LXC containers

When deployed into a container, the following plugins are disabled:

  • cpu, disk, diskio, mem, swap, system, bcache, bond, cgroup, kernel_vmstat

On very busy environments with lots of LXC containers per host, some metrics collected by default may contribute to performance issues with lxcfs. In order to reduce this, you may wish to consider disabling the following plugins:

  • net, netstat, conntrack, nstat

Configuration

Installation method: deb or snap

The charm can deploy using either a deb (default) or a snap installation. juju config telegraf install_method=deb # to use the debian package juju config telegraf install_method=snap # to use the snap Only the needed package (deb or snap) will be installed.

To change the snap channel: juju config telegraf snap_channel=beta

Prometheus Output

By default, Prometheus output is presented available for scraping.

InfluxDB Output

When related to an InfluxDB service, metrics will be pushed to a database named telegraf.

Apache Input

For the apache input plugin, the charm provides the apache relation which uses apache-website interface. Current apache charm disables mod_status and in order to telegraf apache input to work status should be removed from the list of disable_modules in the apache charm config.

Postgresql & MySQL input

Due to relation scope issues, two relations need to be established between telegraf and the postgresql service. first a plain juju-info relation to get standard system metrics and a db-admin relation for PostgreSQL database metrics. eg.:

```
juju add-relation postgresql:juju-info telegraf
juju add-relation postgresql:db-admin telegraf
```

Similarly, for MySQL:

```
juju add-relation mysql:juju-info telegraf
juju add-relation mysql:db-admin telegraf
```

Rabbitmq Input

You will need both relations: * juju add-relation telegraf:juju-info rabbitmq-server:juju-info to start serving data * juju add-relation telegraf:amqp rabbitmq-server:amqp for the actual rabbitmq communication

This relation requires rabbitmq management plugin to be activated: juju set rabbitmq-server management_plugin=true

The rabbitmq charm doesn't appear to give true administrator privileges to relations. On the rabbitmq unit, you will need to grant permissions for the newly created charm user: rabbitmqctl set_permissions -p openstack telegraf-telegraf-XX ".*" ".*" ".*"

The charm configures telegraf to drop idle_since metrics (e.g. idle_since="2018-08-14 0:24:04") which break time series in Prometheus, since any difference in labels (presence or lack, and content) is treated as a new series.

Advanced

Input

A basic set of input plugins are setup by default, which can be overriden with the inputs_config configuration option. Both of these accept raw telegraf configuration.

To configure any of the (default or via relation) plugins, the extra_options charm config can be used. It's yaml, that needs to be encoded as a string in the config. For example:

```yaml
telegraf:
# The trailing | means the following indented section is all one string.
extra_options: |
    inputs:
        cpu:
            percpu: false
            fielddrop: ["time_*"]
        disk:
            mount_points: ["/"]
            ignore_fs: ["tmpfs", "devtmpfs"]
        elasticsearch:
            local: false
            cluster_health: true
        postgresql:
            databases: ["foo", "bar"]
            tagpass:
                db: ["template", "postgres"]
    outputs:
        influxdb:
            precision: ms
```

This extra options will only be applied to plugins configured via relations and the following list (defined in templates/base_inputs.conf): - inputs.cpu - inputs.disk - inputs.diskio - inputs.mem - inputs.net - inputs.netstat - inputs.swap - inputs.system

Output

The only output plugin supported via relation is influxdb, any other output plugin needs to be configured manually (via juju set).

To use a different metrics storage such as Graphite, the raw telegraf plugin configuration needs to be set as a string in outputs_config configuration. For example:

```yaml
telegraf:
  # The trailing | indicates a multiline string in yaml.
  outputs_config: |
    [[outputs.graphite]]
    servers = ["10.0.3.231:2003"]
    prefix = "juju_local.devel.telegraf"
    timeout = 10
```

This will make telegraf agents to send the metrics to the graphite instance.

Contact Information

Configuration

collect_iptables_metrics
(boolean) Enable the collection of iptables metrics, using the iptables telegraf input plugin. Collecting these metrics requires sudo access - enabling this option will install an appropriate, locked-down sudoers file.
collection_jitter
(string) Collection jitter is used to jitter the collection by a random amount. Each plugin will sleep for a random time within jitter before collecting. This can be used to avoid many plugins querying things like sysfs at the same time, which can have a measurable effect on the system.
5s
cpu_idle
(int) Percent of idle CPU at which to generate a Prometheus alert. Default 10%, i.e. alert if the CPU has less than 10% CPU running idle therefore 90% used by us+sy+wa+st.
10
debug
(boolean) Run telegraf in debug mode
disabled_plugins
(string) Colon(:) separated plugins names to disable, including builtin input plugins and custom exec metrics in `files/telegraf_exec_metrics.py`, e.g.: cpu:mem:buddyinfo:zoneinfo
extra_options
(string) YAML with extra options for out|inputs managed by relations or in the default config. example: inputs: cpu: percpu: false fielddrop: ["time_*"] disk: mount_points: ["/"] ignore_fs: ["tmpfs", "devtmpfs"] elasticsearch: local: false cluster_health: true postgresql: databases: ["foo", "bar"] tagpass: db: ["template", "postgres"] outputs: influxdb: precision: ms
extra_packages
(string) Space separated list of extra deb packages to install.
extra_plugins
(string) Extra plugins, manually configured. This is expected to be a string and will be saved "as is" in telegraf.d/extra_plugins.conf
flush_interval
(string) Default data flushing interval for all outputs. You should not set this below interval. Maximum flush_interval will be flush_interval + flush_jitter
10s
flush_jitter
(string) Jitter the flush interval by a random amount. This is primarily to avoid large write spikes for users running a large number of telegraf instances. ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
5s
hostname
(string) Override default hostname. {model} is replaced with the Juju model name and {unit} is replaced with the sanitized unit name (eg. service_name-0). {uuid} is replaced by the model UUID, for sites without unique model names. {host} is replaced by the machine hostname
{model}:{unit}
inputs_config
(string) [inputs.xxx] sections as a string, this override default input plugins.
install_keys
(string) List of signing keys for install_sources package sources, per charmhelpers standard format (a yaml list of strings encoded as a string). The keys should be the full ASCII armoured GPG public keys. While GPG key ids are also supported and looked up on a keyserver, operators should be aware that this mechanism is insecure. null can be used if a standard package signing key is used that will already be installed on the machine, and for PPA sources where the package signing key is securely retrieved from Launchpad.
- null
install_method
(string) Installation method. Valid values are "deb" or "snap".
deb
install_sources
(string) List of extra apt sources, per charm-helpers standard format (a yaml list of strings encoded as a string). Each source may be either a line that can be added directly to sources.list(5), or in the form ppa:<user>/<ppa-name> for adding Personal Package Archives, or a distribution component to enable.
- ppa:telegraf-devs/ppa
interval
(string) Default data collection interval for all plugins
10s
lead_time
(string) Time period to use when alerting on predictive Prometheus rules, e.g a disk might run low on space in $lead_time, based on the previous usage over lead_time. Default 14d, i.e. based on the previous 14 days, alert if we have 14 days disk left. String, e.g. 14d, 4w.
14d
logfile
(string) Name of the file to be logged to when using the file logtarget. If set to the empty string then logs are written to stderr.
/var/log/telegraf/telegraf.log
metric_buffer_limit
(int) Telegraf will cache metric_buffer_limit metrics for each output, and will flush this buffer on a successful write.
10000
nagios_context
(string) A string that will be prepended to instance name to set the host name in nagios. So for instance the hostname would be something like: juju-myservice-0 If you're running multiple environments with the same services in them this allows you to differentiate between them.
juju
nagios_servicegroups
(string) Comma separated list of nagios servicegroups for the telegraf check
juju
outputs_config
(string) [outputs.xxx] sections as a string
package_status
(string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".
install
prometheus_context
(string) A string that will be used to identify this site in the 'cloud_name' label in Prometheus. If you're running multiple environments with the same services in them this allows you to differentiate between them.
juju
prometheus_datasource
(string) Specifies the datasource for Grafana dashboards. This usually corresponds to the application name of the related Prometheus. Defaults to "prometheus". Note: The complete datasource inserted into dashboard will be "$prometheus_datasource - Juju generated source" This matches how the Prometheus2/Grafana charms format the datasource.
prometheus
prometheus_ip_range
(string) If set, the comma-separated list of IP Ranges which are allowed to access metrics. ex: prometheus_ip_range = "192.168.0.0/24,192.168.1.0/30"
prometheus_output_port
(string) If set prometheus output plugin will be configured to listen on the provided port. If set to string "default" the charm will use default port (9103)
9103
promreg_authtoken
(string) The Prometheus Registration authtoken.
promreg_url
(string) The http(s) url and port where the Prometheus Registration service is listening.
quiet
(boolean) Run telegraf in quiet mode
round_interval
(boolean) Rounds collection interval to 'interval' ie, if interval="10s" then always collect on :00, :10, :20, etc.
True
snap_channel
(string) Snap Store channel from which to install the telegraf snap.
stable
snapd_refresh
(string) How often snapd handles updates for installed snaps. The default (an empty string) is 4x per day. Set to "max" to check once per month based on the charm deployment date. You may also set a custom string as described in the 'refresh.timer' section here: https://forum.snapcraft.io/t/system-options/87
socket_listener_port
(string) If set, the input socket_listener will listen on the specified port. The default option is to listen to 8094.
8094
tags
(string) Comma separated list of global tags. ie, 'dc=us-east-1,rack=1a' will tag all metrics with dc=us-east-1 and rack=1a
wait_time
(string) When exporting Prometheus alerting rules, how long a condition should be true before firing an alert. String, e.g. 5m, 1h, etc.
5m