Apache® Kafka® message streams (topics) can be replicated between brokers. A replica is intended to provide a real-time copy of the entire partition of a given topic. The replica provides redundancy and protects against broker failure. The clients can fail over gracefully to the replica hosted on another broker. Multiple replicas may be set up, each duplicating the content of the entire partition (and all the needed disk space).

Each partition has a leader server and one or more follower, or replica, servers. Leaders handles all the read and write requests from consumers and producers for a topic partition. Followers consume messages from the leader and take over if a leader dies.

In Kafka, when the client sends a message to the leader, the leader responds with an acknowledgment that the message has been successfully stored. The client may specify the number of acknowledgments that it requires from the broker and its replicas, before the broker receiving the message acknowledges the client's request. There are three options for acknowledgments. The client may send messages in an unreliable manner, by indicating it needs no acknowledgments. The client can also indicate if it only wants an acknowledgment when the leader stores the message. The third, and most reliable, option is for the client to indicate it wants an acknowledgment that the leader and all its followers successfully stored the message.

If the leader broker is unavailable to the client, assuming it has the address information of other followers, the client fails over to a replica broker and continues sending messages to the new broker. The replica becomes the new leader. If the new broker has a complete up-to-date replica, then no messages are lost.

The combination of multiple partitions split across multiple brokers, and with each partition replicated across the brokers, provides a very high level of confidence that messages can be sent and stored reliably with minimal interruption, or coding on the client.

Much of this process is a matter of selecting the appropriate configuration based on the reliability needs of the application.