Event-streaming platforms, such as Apache® Kafka®, were created in order to overcome the problem with continuous streams of data. Kafka's solution provides these capabilities:
  • Allows you to publish and subscribe to streams of events
  • Allows you to easily store these streams for as long, or short, as you want
  • Allows you to process these events in real-time or defer processing until later

Event-streaming platforms have common characteristics. They are single platforms that can connect anyone to each event, provide a real-time stream of events, and provide a historical view of events.

With Kafka, there are senders (producers) and receivers (consumers) of events (also known as messages).

Servers and clients that communicate over TCP make up Kafka's distributed system. A Kafka cluster is made up of one or more servers. A Kafka broker is a type of server which stores the message (event) streams, persistently. Clients are applications or microservices that read, write, and process streams of events in a fault-tolerant manner. Kafka is highly scalable. You use partitions to scale your system.

Event-streaming platforms are used across a wide-variety of industries and have many use cases, such as:
  • Real-time fraud detection
  • Payment and financial transaction processing
  • Hospital patient monitoring
  • Transportation monitoring and tracking
  • Real-time e-commerce
  • Messaging