prometheus grok exporter #2

Supports: focal bionic xenial
Add to new model

Description

Export Prometheus metrics from arbitrary unstructured log data.


Juju prometheus-grok-exporter charm

grok_exporter allows to export Prometheus metrics from arbitrary unstructured log data.

This charm can be related to the prometheus2 charm, to nrpe charm
and to grafana

Build

Optionally export the CHARM_BUILD_DIR env var or the default /tmp/charm-builds will be used.

To build the charm run:

make build

Configuration

Using the default configuration of the charm, prometheus-grok-exporter won't start until the metrics and global paths are
set.

  • global_paths is comma separated list of files to monitor. All metrics not specifying an individual path
    are applied to all log files defined by this option.

  • metrics is a yaml file defining valid metrics for the prometheus-grok-exporter.
    Refer to the metrics documentation
    from the grok_exporter git repository.

Example of a valid configuration:

  prometheus-grok-exporter:
    options:
      global-paths: "/var/log/syslog"
      metrics: |
        metrics:
        - type: counter
          name: syslog_msgs_total
          help: Total number of syslog msgs, by program (ie daemon)
          match: '%{SYSLOGBASE}'
          labels:
            program: '{{.program}}'

Deploy

Add as subordinate of an existing charm.

i.e.

    juju deploy cs:ubuntu --series bionic
    juju deploy cs:prometheus2 prometheus --series bionic
    juju deploy cs:prometheus-grok-exporter --series bionic
    juju add-relation ubuntu prometheus-grok-exporter
    juju add-relation prometheus-grok-exporter:prometheus-client prometheus:target
    juju add-relation prometheus:grafana-source grafana:grafana-source
    juju add-relation prometheus-grok-exporter grafana

Grafana Dashboard

The charm will send a default dashboard to Grafana stored in files/grafana-dashboards/grok-exporter-dashboard.json.
The dashboard assumes that in Grafana is set a datasource called prometheus - Juju generated source.

If the datasource is called differently in grafana (i.e. prometheus2 - Juju generated source),
you will need to provide a new dashboard through Grafana.

Then you can proceed with deleting the old dashboard manually from the Grafana web interface.

Testing

Juju should be installed and bootstrapped on the system to run functional tests.

    export MODEL_SETTINGS=<semicolon-separated list of "juju model-config" settings>
    make test

NOTE: If you are behind a proxy, be sure to export a MODEL_SETTINGS variable as
described above. Note that you will need to use the juju-http-proxy, juju-https-proxy, juju-no-proxy
and similar settings.

Contact Information


Configuration

enable-autodiscovery
(boolean) By default, grok exporter will automatically expose the metrics defined in files/metrics-mapping.yaml if the log files they are referred exist on the unit. To prevent this mechanism, set this option to false.
True
fail-on-missing-logfile
(boolean) Grok exporter will not start if the log files to monitor don't exist if this is set to true.
True
global-paths
(string) Comma separated list of files to monitor. All metrics not specifying an individual path are applied to all log files defined by this option.
grok-patterns-path
(string) Directory with the pre-defined grok patterns. By default the charm will use the grok patterns shipped by the snap from https://github.com/logstash-plugins/logstash-patterns-core
/var/snap/prometheus-grok-exporter/current/logstash-patterns-core/patterns
listen-address
(string) The address grok-exporter will listen on to expose the metrics.
0.0.0.0
metrics
(string) YAML formatted list of metric definitions, specifying how log lines are mapped to Prometheus metrics. For documentation refer to https://github.com/fstab/grok_exporter/blob/master/CONFIG.md#metrics-section Example of valid confinguration: metrics: - type: counter name: syslog_msgs_total help: Total number of syslog msgs, by program (ie daemon) match: '%{SYSLOGBASE}' labels: program: '{{.program}}' - type: counter name: syslog_msgs_errors_total help: Total number of syslog msgs with errors, by program (ie daemon) match: '%{SYSLOGBASE}.*[Ee]rrors?(?!${WORD})' labels: program: '{{.program}}'
nagios_context
(string) A string that will be prepended to instance name to set the host name in nagios. So for instance the hostname would be something like: juju-myservice-0 If you're running multiple environments with the same services in them this allows you to differentiate between them.
juju
nagios_servicegroups
(string) Comma separated list of nagios servicegroups for the graphite check.
juju
port
(int) The port grok-exporter will listen on to expose the metrics.
9144
promreg_authtoken
(string) The Prometheus Registration authtoken.
promreg_url
(string) The http(s) url and port where the Prometheus Registration service is listening.
snap-channel
(string) Controls the snap channel name. Supported values are: "stable", "candidate", "beta" and "edge"
stable
snap_proxy
(string) DEPRECATED. Use snap-http-proxy and snap-https-proxy model configuration settings. HTTP/HTTPS web proxy for Snappy to use when accessing the snap store.
snap_proxy_url
(string) DEPRECATED. Use snap-store-proxy model configuration setting. The address of a Snap Store Proxy to use for snaps e.g. http://snap-proxy.example.com
snapd_refresh
(string) How often snapd handles updates for installed snaps. The default (an empty string) is 4x per day. Set to "max" to check once per month based on the charm deployment date. You may also set a custom string as described in the 'refresh.timer' section here: https://forum.snapcraft.io/t/system-options/87