To address big data challenges (described earlier in this chapter), there needs to be a data processing architecture to handle massive quantities of data to process rapidly with batch processing and stream processing methods.

Some fundamental characteristics of the Lambda architecture are as follows:

• It is dependent on underlying data principles of append-only, immutable, and atomic
• It thrives on balancing latency, throughput, and fault-tolerance
• It correlates with the growth of big data and real-time analytics
• It helps to mitigate the latencies of MapReduce

The preceding diagram depicts the Lambda architecture with three primary layers called the batch processing layer, the speed or real-time processing layer, and serving layers for responding to queries.

The three primary layers are explained here:

• Batch layer: This precomputes results, using a distributed processing system output to the read-only data store, and updates views by replacing the existing precomputed views. Data accuracy in the views is high with batch jobs (accuracy over latency).
• Speed/Real-time layer: This processes data streams in real time and the views are almost instantaneous, but maybe with less data accuracy (latency over accuracy). However, those views can be updated later by batch methods (accuracy over latency).
• Serving layer: This stores outputs from the batch and speed layers to respond to ad-hoc queries either by precomputed views or new views from the processed data.