History of Lambda Architecture

Nathan Marz coined the term Lambda Architecture (LA) to describe a generic pattern for data processing that is scalable and fault-tolerant. He gathered this expertise working extensively with big-data-related technologies at BackType and Twitter. The pattern is conceptualized to handle/process a huge amount of data by using two of its important components, namely batch and speed layer. Nathan generalized his findings and experience in the form of this pattern, which should cater to some of the important architecture principles, such as these:

• Linear scalable: It should scale out and not up and should cater to different kinds of use cases
• Fault-tolerant: Capable of a wide range of workloads, it should also shield the system from hardware and software failure and inherent human mistakes
• Backtype: Reads and updates
• Extensibility: Manageable, easy to extend, and easy to keep adding new features and data elements

There is a wealth of details documented at http://lambda-architecture.net/. The architecture pattern became significant with the emergence of big data and the enterprises' focus on real-time analytics and digital transformation. Why the pattern was named Lambda (symbol λ) is not very clearly detailed.

However, we have always thought the greek letter in association with the pattern to be a way by which data comes from two places. Batch and Speed data (layer) pipeline represents the curved parts of the lambda Symbol, which then combine and serve through the serving layer (the line merging from the curved part), as shown in the following figure:

Figure 01: The Lambda Symbol