The Kafka message broker, which has been further developed by the community and Confluent under the umbrella of the Apache Software Foundation, is officially available in version 2.7. The update of the open source project, which is designed for real-time processing of large amounts of data, contains numerous bug fixes, various improvements and some new functions. The Kafka team has made progress especially in efforts to replace ZooKeeper with a self-managed metadata quorum with a view to Release 3.0 (KIP – 500).

Apache Kafka 2.7 offers a new Inter-Broker API for this purpose. It should create the prerequisites for making changes in the In-sync Replica (ISR) faster and more safely. The previous approach, updates of the controller (broker responsible in the Kafka cluster for the status management of partitions and replicas) and the partition leaders in ZooKeeper (znode “https://www.heise.de/brokers/topics/ / partitions / [partitionId] / state “) saved information about zkVersion was accompanied by time delays. In particular, the forwarding of zkVersion by the leader to the controller takes place up to 60 seconds delayed. To prevent future metadata queries from delivering outdated information, the new AlterIsr – API help. It grants the controller the exclusive privilege to update Partition Leader and ISR.

Bill Bejeck from the Developer Relations Team Confluent gives a compact overview of Apache Kafka 2.7

in the video Another measure that should help to replace ZooKeeper is the introduction of a broker-side SCRAM configuration API. As stated in the Kafka Improvement Proposal KIP – 554 in front seen, SCRAM credentials can in future be managed via the Kafka protocol. Version 2.7 of the associated tool kafka-configs has been adapted to the new protocol API.

In the update of the Message Broker there is also a Core Raft implementation in accordance with KIP- 595. A separate “Raft” module contains the core of the consensus protocol. As long as the integration with the controller has not yet been completed, however, Kafka users have a separate server available to test the performance of the Raft implementation.

More details and a complete one The Confluent blog post announcing the new version and the release notes for Apache Kafka 2.7 provide an overview of all new features.

(map)