You should be able to extract valuable, meaningful information (insights) from the enormous volumes of data to improve an organization's decisions that involve various challenges, such as data regulations, faster decisions, interactions with customers, dealing with legacy systems, disparate data sources, and so on. So, to address all those challenges efficiently, researchers came up with a unified architecture consisting of layers at different levels:
The preceding pyramid depicts the significant attributes of big data layers and the problems that are addressed in each layer. As we have mentioned earlier, big data is not a single technology or a framework solving just a set of use cases; it is a set of tools, processes, technologies, and a system infrastructure that helps businesses to make much smarter analysis and take smarter decisions based on the massive volume of data traces.
Unified big data architecture consists of various layers. It provides a way to organize different components to address problems and it represents unique functions:
- Big data sources: Data coming from several channels, such as handheld devices, software applications, sensors, legacy databases, and so on
- Data messaging and storage: Acquires data from the data sources, data compliance, and storage formatting
- Data analysis: Data model management, analytics engines, and access to data message stores
- Data consumption: Dashboards, insights, reporting, and so on
The preceding diagram depicts different levels and layers of the big data landscape. These layers perhaps may be considered as a summary of our earlier introductions of big data concepts and the realization of values in each layer.
Before we look at patterns, let's summarize the big data architecture principle as follows:
- Decoupled data bus
- Right tool usage for the job
- Data structure, latency, throughput, access patterns
- Lambda architecture
- Immutable logs, batch/speed/serving layer
- Cloud-based infrastructure
- System maintenance with low or no admin
- Cost-effective