kafka #26

Supports: bionic
Add to new model

Description

This charm deploys a version of Kafka from user defined PPA.
It's intended for some internal projects and is not supported for general
community use.


Description

Kafka is a high-performance, scalable, distributed messaging system.
This charm provides version 2.2 of the Kafka application from Apache.

Overview

Apache Kafka is an open-source message broker project developed by the Apache
Software Foundation written in Scala. The project aims to provide a unified,
high-throughput, low-latency platform for handling real-time data feeds. Learn
more at kafka.apache.org.

This charm deploys version 2.2 of the Kafka component.

Deploying

This charm requires Juju 2.7 or greater. If Juju is not yet set up, please
follow the getting-started instructions prior to deploying this charm.

Kafka requires the Zookeeper distributed coordination service. Deploy and
relate them as follows:

juju deploy kafka
juju deploy zookeeper
juju deploy easyrsa
juju add-relation kafka zookeeper
juju add-relation kafka easyrsa

Network-Restricted Environments

Charms can be deployed in environments with limited network access. To deploy
in this environment, configure a Juju model with appropriate proxy and/or
mirror options. See Configuring Models for more information.

Using

Once deployed, there are a number of actions available in this charm.

List the zookeeper servers that our kafka brokers
are connected to. The following will list : information for each
zookeeper unit in the environment (e.g.: 10.0.3.221:2181).

juju run-action kafka/0 list-zks
juju show-action-output <id>  # <-- id from above command

Create a Kafka topic with:

juju run-action kafka/0 create-topic topic=<topic_name> \
 partitions=<#> replication=<#>
juju show-action-output <id>  # <-- id from above command

List topics with:

juju run-action kafka/0 list-topics
juju show-action-output <id>  # <-- id from above command

Write to a topic with:

juju run-action kafka/0 write-topic topic=<topic_name> data=<data>
juju show-action-output <id>  # <-- id from above command

Read from a topic with:

juju run-action kafka/0 read-topic topic=<topic_name> partition=<#>
juju show-action-output <id>  # <-- id from above command
juju run-action kafka/0 perf-test topic=one messages=10000 recordsize=10

Verifying Status

Kafka charms provide extended status reporting to indicate when they
are ready:

juju status

This is particularly useful when combined with watch to track the on-going
progress of the deployment:

watch -n 2 juju status

The message column will provide information about a given unit's state.
This charm is ready for use once the status message indicates that it is
ready.

Smoke Test

This charm provides a smoke-test action that can be used to verify the
application is functioning as expected. The test will verify connectivity
between Kafka and Zookeeper, and will test creation and listing of Kafka
topics. Run the action as follows:

juju run-action --wait kafka/0 smoke-test

Watch the progress of the smoke test actions with:

Performance Test

This charm provides a perf-test action that can be used to run the
application performance. The test will create the topic and run the
performance test and log all the data in kafka juju log files.
Run the action as follows:

juju run-action kafka/0 perf-test topic=one messages=10000 recordsize=10
juju debug-log --include kafka/0 would print the test metrics as well.

Scaling

Expanding a cluster with many brokers is as easy as adding more Kafka units.
Expanding cluster does not make any effect on the curernt topics but any new
topic will be distributed on new kafka cluster machine.

juju add-unit kafka

After adding additional brokers, topics may be created with
replication up to the number of ready units. For example, if there are two
ready units, create a replicated topic as follows:

juju run-action kafka/0 create-topic topic=my-replicated-topic \
     partitions=1 replication=2

Query the description of the recently created topic:

juju run --unit kafka/0 'kafka-topics --describe \
    --topic my-replicated-topic --zookeeper <zookeeperip>:2181'
Topic: my-replicated-topic PartitionCount:1 ReplicationFactor:2 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0

Connecting External Clients

By default, this charm does not expose Kafka outside of the provider's network.
To allow external clients to connect to Kafka, first expose the service:

juju expose kafka

Next, ensure the external client can resolve the short hostname of the kafka
unit. A simple way to do this is to add an /etc/hosts entry on the external
kafka client machine. Gather the needed info from juju:

$ juju run --unit kafka/0 'hostname -s' 
kafka-0
$ juju status --format=yaml kafka/0 | grep public-address
public-address: 40.784.149.135

Update /etc/hosts on the external kafka client:

$ echo "40.784.149.135 kafka-0" | sudo tee -a /etc/hosts

The external kafka client should now be able to access Kafka by using
kafka-0:9092 as the broker.

Network Spaces

In some network environments, kafka may need to be restricted to
listen for incoming connections on a specific ingress interface
(e.g.: for security reasons). To do so, kafka charm binds kafka with a
seperate space specifying a ingress address.

Below juju command will give you the ip address where kafka binds to for
ingreee address

juju run --unit kafka/1 "network-get --ingress-address --format yaml listener"

Configuration

auto_create_topics
(boolean) Enable auto creation of topic on the server (auto.create.topics.enable)
True
default_partitions
(int) The default number of log partitions per topic (num.partitions)
1
default_replication_factor
(int) Default replication factor for automatically created topics (default.replication.factor)
1
extra_config
(string) Base 64 encodede string for server properties tuning file.
"bnVtLm5ldHdvcmsudGhyZWFkcz0zCm51bS5pby50aHJlYWRzPTgKc29ja2V0LnNlbmQuYnVmZmVy LmJ5dGVzPTEwMjQwMApzb2NrZXQucmVjZWl2ZS5idWZmZXIuYnl0ZXM9MTAyNDAwCnNvY2tldC5y ZXF1ZXN0Lm1heC5ieXRlcz0xMDQ4NTc2MDAKbnVtLnJlY292ZXJ5LnRocmVhZHMucGVyLmRhdGEu ZGlyPTEKb2Zmc2V0cy50b3BpYy5yZXBsaWNhdGlvbi5mYWN0b3I9MQp0cmFuc2FjdGlvbi5zdGF0 ZS5sb2cucmVwbGljYXRpb24uZmFjdG9yPTEKdHJhbnNhY3Rpb24uc3RhdGUubG9nLm1pbi5pc3I9 MQpsb2cucmV0ZW50aW9uLmhvdXJzPTE2OApsb2cuc2VnbWVudC5ieXRlcz0xMDczNzQxODI0Cmxv Zy5yZXRlbnRpb24uY2hlY2suaW50ZXJ2YWwubXM9MzAwMDAwCnpvb2tlZXBlci5jb25uZWN0aW9u LnRpbWVvdXQubXM9NjAwMApncm91cC5pbml0aWFsLnJlYmFsYW5jZS5kZWxheS5tcz0wCgoK"
extra_packages
(string) Space separated list of extra deb packages to install.
install_keys
(string) List of signing keys for install_sources package sources, per charmhelpers standard format (a yaml list of strings encoded as a string). The keys should be the full ASCII armoured GPG public keys. While GPG key ids are also supported and looked up on a keyserver, operators should be aware that this mechanism is insecure. null can be used if a standard package signing key is used that will already be installed on the machine, and for PPA sources where the package signing key is securely retrieved from Launchpad.
install_sources
(string) List of extra apt sources, per charm-helpers standard format (a yaml list of strings encoded as a string). Each source may be either a line that can be added directly to sources.list(5), or in the form ppa:<user>/<ppa-name> for adding Personal Package Archives, or a distribution component to enable.
inter_broker_protocol_version
(string) Specify the message format version the broker will use to append messages to the logs. If set to empty string, the current broker version is used. This setting is useful for coordinating kafka broker upgrades.
kafka_jmx_opts
(string) options to be used with JMX exporter.
-Djava.rmi.server.hostname=localhost -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
kafka_jmx_port
(string) Specify the port number of jmx server where clients can retries the data.
9999
log_dir
(string) This option will use the driectory for logs. In case of MAAS deployment we should preconfigured the mount point.
/var/lib/kafka/logs
log_message_format_version
(string) Specify the message format version the broker will use to append messages to the logs. If set to empty string, the current broker version is used. This setting is useful for coordinating kafka broker upgrades.
nagios_avg_network_processor_idle_crit
(string) The critical threshold for average idle percentage of a network processor
.10
nagios_avg_network_processor_idle_warn
(string) The warning threshold for average idle percentage of a network processor
.20
nagios_consumer_fetch_time_crit
(string) The critical threshold for the 99th percentile time it takes for a consumer to fetch a message.
500
nagios_consumer_fetch_time_warn
(string) The warning threshold for the 99th percentile time it takes for a consumer to fetch a message.
50
nagios_context
(string) Used by the nrpe subordinate charms. A string that will be prepended to instance name to set the host name in nagios. So for instance the hostname would be something like: juju-myservice-0 If you're running multiple environments with the same services in them this allows you to differentiate between them.
juju
nagios_leader_election_rate_crit
(string) The critical threshold for the leader election rate and latency in ms.
1000
nagios_leader_election_rate_warn
(string) The warning threshold for the leader election rate and latency in ms.
100
nagios_producer_time_crit
(string) The critical threshold for the 99th percentile time it takes to produce a message.
500
nagios_producer_time_warn
(string) The warning threshold for the 99th percentile time it takes to produce a message.
50
nagios_servicegroups
(string) A comma-separated list of nagios servicegroups. If left empty, the nagios_context will be used as the servicegroup
package_status
(string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".
install
port
(int) Port to run the Kafka services on
9093
service_environment
(string) This option will define the kafka service environment will into service section.
-Xmx1G -Xms128M
service_parameter
(string) This option will define the extra kafka service parameters into service section.
LimitNOFILE=128000
ssl_ca
(string) Base64-encoded certificate authority. This CA is used in conjunction with keystone https endpoints and must, therefore, be the same CA used by any endpoint configured as https/ssl.
ssl_cert
(string) Base64-encoded SSL certificate to install and use for Horizon. . juju set kafka ssl_cert="$(cat cert| base64)" \ ssl_key="$(cat key| base64)"
ssl_key
(string) Base64-encoded SSL key to use with certificate specified as ssl_cert.
ssl_key_password
(string) SSL key password.
subject_alt_names
(string) Additional subject alternate names (SANs) that should be added to the server certificate requested from the CA. Specified as a comma-separated list of hostnames and/or IP addresses.