Differencing stream processing and batch processing
The popularity of stream data platforms has been increasing significantly in recent times, due to the requirement of real-time access to information. Enterprises are transitioning parts of their data infrastructure from traditional batch processing to streaming paradigm due to changing business needs and the need to get of Real-Time Insights on data as business events occur.
It's critical to understand the fundamental differences between stream and batch processing:
The following list provides a subset of examples where a streaming data analytics solution can add value to a business:
- An online social media news publisher harvests streams of clickstream to aggregate and enriches the data with demographic data, to deliver relevancy and enhanced news experience to its audience.
- Real-time weather and traffic updates.
- Fraud prevention and detection on financial and non-financial transactions.
- Enhancing customer experience for an online retailer, food delivery, transportation and multitude of online businesses.
- Sensors in connected vehicles, farm machinery, heavy machinery and mechanical devices send data to a streaming application. The application monitors performance detects any potential defect in advance and places orders for required servicing (predictive maintenance), notifies field personal, service center to reach out to the customer. This enables manufacturers to become proactive rather than reactive.
Stream processing brings in a number of business advantages, like the following:
- Translate Real-Time Insights into a compelling customer experience.
- Business Activity Monitoring (BAM). Many large organizations have deep and complex processes. These processes need to be reconciled. These reconciliations are done using batch processing. With Stream Analytics, reconsolidations can happen in real time and this speeds up the business decision process.
- Provides more opportunity for traditional business to enhance and add value to their existing services. For example, if you are heavy equipment manufacture or servicing enterprise by introducing sensors and telemetry into the equipment, you can tap health of the machinery equipment in real time. This will enhance the value of the equipment and opens up net new opportunities to grow.
In the next sections, we will review components that go into designing real-time streaming solution and end the chapter with a canonical architecture. Let's start working on the concepts.