Although the development field of data analytics is not new, it has become more critical than ever as it experiences prodigious quantities of data generated by businesses, sensors, applications, and so on. Once the generated data gets stored, it can give extraordinary insights and helps not only business enterprises but also government and non-government enterprises, social communities, the economy, and much more.
In current technology trends, big data has been involved in many evolutions, from just buzzwords to crunching data from machine learning algorithms. With the exponential explosion of high velocity, high volume, high variety, and the veracity of data sources and streams (the four V's), big data has become the inevitable representative of the architectures, tools, and technologies that handle enterprises increasingly demanding requirements.
In this chapter, we have gone through a brief introduction of the four V's of big data, data analysis technology, and concepts. We also touched upon the big data life cycle and how it helps different stakeholders to achieve and realize their data insights. A brief section covered big data landscapes, and the data layers, as well as most of the architectural patterns associated with big data, involving data pipelines: that is an ordered combination of data acquisition, integration, ingestion, fast processing, storage, rapid access, and analytics stages.
The most crucial theme of this book is architectural patterns, and this chapter reflects it in its big data architecture, and design patterns section, in a sequence of architecture patterns, such as MapReduce, Lambda, and data lake. Then we have covered most common big data (application) design patterns by layers: that is patterns in various big data architectural layers, such as data sources and the ingestion layer, the data storage layer, the data access layer, the data discovery and analysis layer, and the data visualization layer.
Covering big data architectural patterns in one chapter has been very challenging for us, and we have tried our best by providing samples of big data concepts and the most common patterns that help data architects and other data technology stakeholders. We hope this chapter provides them with a head start on their big data journey. As mentioned in many places across this chapter, we strongly encourage readers to refer to the citations section should they need to get exclusive patterns and details of implementations.