HOWTO expand a kafka cluster
Why do that¶
On a mutualized or dedicated platform, you may want to increase the capacity of the platform. For a kafka cluster, it can be to :
- increase the retention time
- increase the EPS rate
Prerequisites¶
- To have access to the deployment device (Ansible VM)
- To have the new kafka devices ready for production (repository, update, DNS, LDAP etc...)
- If you are in production, you must have perform a RFC
What to do¶
- Check the current deployment from the deployment device.
1 | $ punchplatform-deployer.sh --deploy -Kk --check --diff -t kafka -l kafka_servers |
Note
You can also limit the deployment to a specific kafka_cluster by using
kafka-
- Create a tag to the configuration before the extension.
1 2 | $ cd $PUNCHPLATFORM_CONF_DIR $ git tag -a vX.Y -m "X.Y before extension" |
- Update PunchPlatform properties with the new kafka devices (brokers).
1 2 3 4 5 6 7 8 | "kafka" : { "clusters" : { "local" : { "brokers" : ["node01:9092", "node02:9092", "node03:9092"], ... } } } |
Warning
Do not forget to write TCP port in the list !!!!
- Deploy Metricbeat on all new device first
1 | $ punchplatform-deployer.sh --deploy -Kk -t metricbeat -l kafka_servers |
Check in Grafana Systeme Dashboard the system metrics of the new nodes.
- Deploy Kafka on only one new node.
1 | $ punchplatform-deployer.sh --deploy -Kk -t kafka -l <hostname> |
- Check the new ID of the new broker in zookeeper by using the zookeeper debug tool
1 2 | $ punchplatform-zookeeper-console.sh $ ls /<platform_id>/kafka-<kafka_cluster>/brokers |
You must have the ID of the new node. If not, please contact the PunchPlatform Support.
- Deploy Kafka on all nodes.
1 | $ punchplatform-deployer.sh --deploy -Kk -t kafka -l kafka_servers |
After this step, the kafka cluster has rebooted. So check the current processing of your platform. Especially the writing in this kafka cluster.
- Deploy again to check the current deployment.
1 | $ punchplatform-deployer.sh --deploy -Kk -t kafka -l kafka_servers |
-
Perform last check.
-
Check the current processing of your platform. No backlog must be observed.
-
Check the kafka tab in the PunchPlatform Admin Server (port 5000)
-
End of extension.
Tag the configuration after the operation.
1 2 | $ cd $PUNCHPLATFORM_CONF_DIR $ git tag -a vX.Y -m "X.Y after extension" |