filebeat #2
Description
As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis.
- Tags:
- logging ›
- monitoring ›
Overview
Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis.
Usage
Filebeat can be added to any principal charm thanks to the wonders of being a subordinate charm. The following example will deploy an ubuntu log source along with the elk stack so we can visualize our log data.
juju deploy ~elasticsearch-charmers/bundle/elk-stack
juju deploy xenial/filebeat
juju deploy xenial/ubuntu
juju add-relation filebeat:beats-host ubuntu
juju add-relation filebeat logstash
Deploying the minimal Beats formation
If you do not need log buffering and alternate transforms on data that is being shipped to ElasticSearch, you can simply deploy the 'beats-core' bundle which stands up Elasticsearch, Kibana, and the known working Beats subordinate applications.
juju deploy ~containers/bundle/beats-core
juju deploy xenial/ubuntu
juju add-relation filebeat:beats-host ubuntu
juju add-relation topbeat:beats-host ubuntu
Changing what is shipped
By default, the Filebeat charm will ship any container logs present in
/var/lib/docker/containers
as well as everything in:
/var/log/*/*.log
/var/log/*.log
If you'd rather target specific log files:
juju config filebeat logpath=/var/log/mylog.log
Testing the deployment
The applications provide extended status reporting to indicate when they are ready:
juju status
This is particularly useful when combined with watch to track the on-going progress of the deployment:
watch juju status
The message for each unit will provide information about that unit's state. Once they all indicate that they are ready, you can navigate to the kibana url and view the streamed log data from the Ubuntu host.
juju status kibana --format=yaml | grep public-address
Navigate to http://<kibana-ip>/ in a browser and begin creating your dashboard visualizations.
Upgrading filebeat
Upgrades are handled at both the charm and apt repository levels. Use
upgrade-charm
to get the latest charm code on all filebeat units:
juju upgrade-charm filebeat
Apt repositories are scanned any time the install_sources
config changes. If
a new version of filebeat is found in the configured repository, juju status
will instruct operators to run the reinstall
action. This action must be
run on each filebeat unit:
juju run-action --wait filebeat/0 reinstall
The reinstall
action will stop the filebeat service, purge the apt package,
and reinstall the latest version available from the configured repository.
Scale Out Usage
As a subordinate charm, filebeat will scale when additional principal units are
added. For example, adding ubuntu
units that are related to filebeat
will
automatically install and configure filebeat
for the new unit(s).
juju add-unit ubuntu
To monitor additional applications, simply relate the filebeat subordinate:
juju add-relation filebeat:beats-host my-charm
Contact Information
Community / Help
Configuration
- exclude_files
- (string) A list of regular expressions to match the files that you want Filebeat to ignore: https://www.elastic.co/guide/en/beats/filebeat/6.7/filebeat-input-log.html#filebeat-input-log-exclude-files"
- ["/filebeat.*", ".*gz$"]
- exclude_lines
- (string) A list of regular expressions to match the lines that you want Filebeat to exclude: https://www.elastic.co/guide/en/beats/filebeat/6.7/filebeat-input-log.html#filebeat-input-log-exclude-lines"
- []
- extra_inputs
- (string) A YAML list which will be injected to define additional prospectors/inputs.
- extra_packages
- (string) Space separated list of extra deb packages to install.
- fields
- (string) Space seperated list of key:value that the prospector will assign as field to each beat
- harvester_buffer_size
- (int) Defines the buffer size every harvester uses when fetching the file
- 16384
- install_keys
- (string) Elastic Beats apt repository key
- D88E42B4
- install_sources
- (string) Elastic Beats apt repository
- deb https://artifacts.elastic.co/packages/6.x/apt stable main
- kafka_hosts
- (string) A comma separated list of kafka output hosts in addition to those from relations.
- kafka_topic
- (string) Topic name. Format strings are allowed.
- %{[type]}
- kafka_topics
- (string) Expert setting topics filter.
- kafka_version
- (string) Expert setting kafka version.
- kube_logs
- (boolean) Whether or not to include the Kubernetes audit log as well as any K8s metadata when container logs are present on the system: https://www.elastic.co/guide/en/beats/filebeat/6.7/add-kubernetes-metadata.html Note: this option has no effect when related to a non Charmed Kubernetes charm.
- log_level
- (string) Beats log level. One of debug, info, warning, error, or critical: https://www.elastic.co/guide/en/beats/filebeat/5.6/configuration-logging.html#level
- info
- logging_to_syslog
- (boolean) Send beats logs to syslog: https://www.elastic.co/guide/en/beats/filebeat/5.6/configuration-logging.html#_to_syslog
- True
- logpath
- (string) Space separated log paths to monitor. Can contain wildcards.
- /var/log/*.log /var/log/*/*.log
- logstash_hosts
- (string) A comma separated list of logstash output hosts in addition to those from relations.
- logstash_ssl_cert
- (string) Public SSL certificate data (base64 encoded) for connecting securely to logstash.
- logstash_ssl_key
- (string) Private SSL key data (base64 encoded) for connecting security to logstash.
- max_bytes
- (int) Maximum number of bytes a single log event can have. Default 10MB
- 10485760
- package_status
- (string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".
- install