As the next-generation Logstash Forwarder, Filebeat tails logs and quickly
sends this information to Logstash for further parsing and enrichment or to
Elasticsearch for centralized storage and analysis.
A lightweight, open source shipper for log file data. As the
next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this
information to Logstash for further parsing and enrichment or to Elasticsearch
for centralized storage and analysis.
Filebeat can be added to any principal charm thanks to the wonders of being
a subordinate charm. The following usage example will deploy the elk stack,
so we can visualize our log data once we've established the link between
Filebeat and Logstash
juju deploy ~containers/bundle/elk-stack juju deploy ubuntu juju deploy ~containers/trusty/filebeat juju add-relation filebeat:beats-host ubuntu juju add-relation filebeat logstash
Deploying the minimal Beats formation
If you do not need log buffering and alternate transforms on your data thats
being shipped to ElasticSearch you can simply deploy the 'beats-core' bundle
which stands up Elasticsearch, Kibana, and the three known working Beats
juju deploy ~containers/bundle/beats-core juju deploy ubuntu juju add-relation filebeat:beats-host ubuntu juju add-relation topbeat:beats-host ubuntu juju add-relation packetbeat:beats-host ubuntu
A note about the beats-host relationship
The Beats suite of charms leverage the implicit "juju-info" relation interface
which is special and unique in the context of subordinates. This is what allows
us to relate the beat to any host, but may have some display oddities in the
juju-gui. Until this is resolved, it's recommended to relate beats to their
principal services using the CLI
Changing whats being shipped
by default, the Filebeat charm is setup to ship everything in:
If you'd rather target specific log files:
juju set-config filebeat logpath /var/log/mylog.log
Testing the deployment
The services provide extended status reporting to indicate when they are ready:
juju status --format=tabular
This is particularly useful when combined with watch to track the on-going
progress of the deployment:
watch -n 0.5 juju status --format=tabular
The message for each unit will provide information about that unit's state.
Once they all indicate that they are ready, you can navigate to the kibana
url and view the streamed log data from the Ubuntu host.
juju status kibana --format=yaml | grep public-address
open http://<kibana-ip>/ in a browser and begin creating your dashboard visualizations
Scale Out Usage
This bundle was designed to scale out. To increase the amount of log storage and
indexers, you can add-units to elasticsearch.
juju add-unit elasticsearch
You can also increase in multiples, for example: To increase the number of
Logstash parser/buffer/shipping services:
juju add-unit -n 2 logstash
To monitor additional hosts, simply relate the Filebeat subordinate
juju add-relation filebeat:beats-host my-charm
- Charles Butler Chuck@dasroot.net
- Matthew Bruzek email@example.com
- Tim Van Steenburgh firstname.lastname@example.org
- George Kraft email@example.com
- Rye Terrell firstname.lastname@example.org
- Konstantinos Tsakalozos email@example.com
- (string) ElasticSearch index format
- (string) A list of regular expressions to match the files that you want Filebeat to ignore. https://www.elastic.co/guide/en/beats/filebeat/5.3/configuration-filebeat-options.html#exclude-files
- (string) A list of regular expressions to match the lines that you want Filebeat to exclude. https://www.elastic.co/guide/en/beats/filebeat/5.3/configuration-filebeat-options.html#exclude-lines
- (string) Space separated list of extra deb packages to install.
- (string) Space seperated list of key:value that the prospector will assign as field to each beat
- (int) Defines the buffer size every harvester uses when fetching the file
- (string) repository key
- (string) apt repository to fetch beats from
- deb http://packages.elastic.co/beats/apt stable main
- (string) A comma separated list of kafka output hosts in addition to those from relations.
- (string) Topic name. Format strings are allowed. https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_literal_topic_literal
- (string) Expert setting topics filter. https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_literal_topics_literal
- (boolean) Add a prospector to ship logs from Kubernetes pods
- (boolean) Send filebeat logs to syslog https://www.elastic.co/guide/en/beats/filebeat/master/configuration-logging.html#_literal_to_syslog_literal
- (string) Space separated log paths to monitor. Can contain wildcards.
- /var/log/*.log /var/log/*/*.log
- (string) A comma separated list of logstash output hosts in addition to those from relations.
- (string) Public SSL certificate data (base64 encoded) for connecting securely to logstash.
- (string) Private SSL key data (base64 encoded) for connecting security to logstash.
- (int) Maximum number of bytes a single log event can have. Default 10MB
- (string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".