Message from CEO

EVENT STREAMING

Business Decisions must be data-driven

As we talk about product modernisation, which means moving away from legacy products to newer, more efficient architectures, there is a new tech on the block: Event Streaming.

Streaming technologies help process the events and generate insights for immediate actions. Let us take a basic view of the different terminologies for this new technology.

Events

Any change in the state of an object due to a consumer’s actions. Any such change results in data generation. An example of a data point could be a user purchasing an item from an eCommerce website. Here the change is from cart to purchase.

Event Streams

Event streams are sequences of events that have significant data with reference to a particular time.

Let us take an example of 4 services A, B, C, and D. In traditional architecture, A would ask the state of other services and get a response from them for its processing. This process has an inherent risk, i.e., assuming that the other services are always available.

This risk is removed when the other services continuously publish their data (Events). This continuous publishing is called an Event Stream. ‘A’ can now subscribe to this stream and process the data for its use. This publishes/subscribe (pub/sub) nature of using events is called Event Streaming.

It provides real-time data about the state changes. This information can come from IoT devices, mobile apps, cloud services, or other databases. Organizations can use this “Data in motion” to generate valuable insights and take appropriate actions in line with the senses.

Event Stream Processing

This requires two different components: the first is data storage using appropriate technology, and the second is the technology for data processing. The first ensures that the events are stored in chronological order. Apache Kafka is the most used stream engine. It holds the events temporarily and processes them.

Many organizations understand the importance of data for them. They have also started gathering data to find valuable insights. Traditionally, organizations collected data in batches, but it was processed only when sufficient or minimum data was available. This is where Event Stream Processing is advantageous. It processes each single data point in real-time. It also helps in scaling in real-time when data increases. Additionally, it provides insights continuously.

A few use cases for the ESP:

• Supply Chain Tracking: real-time tracking of food delivery agents; continuous tracking of vehicles/ships and their state.
• Financial transactions: to detect any anomaly/fraud transactions
• IoT data: capturing data from sensors like temperature, heat, etc.
• Healthcare: patient health status
• Automotive: real-time car status, driverless cars
• Manufacturing: continuous status from machines for predictive maintenance resulting in a reduction in costs

Event Stream Platforms provide the technology to take large amounts of data and process them or make them available to other applications for real-time insights. They also offer scalable and resilient storage.

There are a few open-source platforms that are commonly used:

• Apache Kafka
• Apache Pulsar

A few significant vendors also provide ESP as a managed service:

• AWS Kinesis
• Azure Event Hubs
• IBM Event Streams

Event Stream Processing is the next big thing that has disrupted the market. It is a potent tool that helps producers come closer to their consumers by taking event analytics to the next level.

Here I am summarizing the top benefits of ESP:

• Decoupling of event publishers and subscribers.
• Higher efficiency of services due to decoupling.
• Real-time feedback and data processing.
• Reliable Platforms for processing.

Overall, as legacy systems make way for newer technologies, ESP will form an essential tool for every product to provide valuable insights to an organization to improve its operations.