Event streams and the unified event log pattern

A unified event log is the collection point/storehouse for all events (state changes, threshold break-ins or any noteworthy deviations, deficiencies, disturbances, and so on) that occurred in any participating microservice. Each participating service can also opt to retain a local log of its own state changes. However, collecting and stocking all kinds of event logs in a single and unified event store is capable of opening up a host of fresh possibilities and opportunities. The complete and 360-degree view of all the events presented in the unified event log can be used to play back selected events and create a projection of the information in any way desired. A variety of data analytics can be done on the event data in real time in order to extract actionable insights. The microservices' performance/throughput, scalability, availability, auditability, security, operational status, and so on can be easily deduced from the event store. The predictive and preventive maintenance of microservices can also be achieved through such a centralized and consolidated event log data.

A unified log can have demanding requirements for performance and scalability given a large number of microservices that can potentially source event streams. Apache Kafka's design for speed, scale, durability, and massive concurrency, together with its model allowing only immutable records to be written to it, makes it an increasingly popular choice as a unified log. Kafka maintains message feeds in distributed and replicated partitions.

Due to the continuous explosion of multifaceted, networked, and embedded devices, the number of events getting generated and captured is growing rapidly. The need is undoubted to have a highly scalable messaging platform that is able to receive a very high number of events emanating from different and distributed sources. We all know that the Apache Kafka messaging platform has the inherent ability to receive millions of events per second. The events are then partitioned so that both batch and real-time processing requirements can be met. Since service architecture patterns call for smart endpoints and dumb pipes, Kafka will do just enough for most application and system integration use cases.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset