Consolidation of Multiple Data Fields

MDM can help eliminate duplicate records by merging them together into a single, consolidated record. This would be great for the scenario shown in Figure 17.2 as the customer records across the three different databases could potentially be consolidated into one database that contains true information about the customer and the relationship of the organization with that customer.

During the consolidation process, MDM leverages a central repository known as a hub (or consolidation hub). A hub is a single source of truth where all the master data from multiple data sources is consolidated. Once the data is in a hub, MDM leverages algorithms for cleansing, matching, and merging to come up with a complete single record—the golden record—that is stored in the hub. After this consolidation, all systems and applications will leverage the golden record instead of pulling master data from various systems.

Building on our example of customer records across multiple databases, Figure 17.3 shows how data is consolidated in MDM (into the hub) and what the golden record looks like.

All systems and applications will now leverage the golden record instead of pulling master data from various systems. The golden records may be used for analytics, report generation, or customer interaction.

Data stewards play an important role in MDM as they can further optimize the output from the algorithms and ensure that the golden records are optimized. (Data stewards were introduced in Chapter 15, “Data Governance Concepts: Ensuring a Baseline.”) Data stewards ensure that proper governance is adhered to when it comes to golden records and look after aspects of life cycle management.

Now, you might ask—how does the field matching and merging process work? The matching process leverages matching rules (or matching groups) to identify similar records across columns and shortlists them for merging. Matching processes might use exact matching or fuzzy matching with similar or nearly similar values. The merging of data fields is driven by the MDM post-matching process, which tags fields with identical data so that values can be merged to arrive at golden records.

At a high level, MDM data consolidation occurs in the following stages:

1. Data loading and initial check: In this stage, the data is loaded from the master database, and the data is checked against a set of predefined rules to ensure that it is clean.

2. Data matching: This stage involves matching the data based on matching rules or matching groups to find duplicates.

3. Finding best records: Best records are created based on matching rules or matching groups.

4. Merging: Data is merged based on matched field values.

5. Validation and promotion: Once the newly created records are validated, they are promoted to become golden records.

Finally, after golden records are created, as mentioned earlier, they are used for analysis and business intelligence reporting. Figure 17.4 illustrates this process.

Standardization of Data Field Names

Organizations typically have standards in place to manage things (including data) in a certain way. An organization that is standardizing data is trying to achieve a consistent and well-defined data format across all business functions. Converting multiple datasets into a common data format helps everyone understand and transact in the same way; however, in the real world, this doesn’t often happen, and many types of data formats coexist. For example, one software vendor might structure its datasets in a unique way; as another example, the fields that different datasets cover may vary.

Let’s consider an example of how data standardization can bring harmony to the way data is managed in an organization. Consider a sales customer database that captures customer details such as:

• Name

• Address

• Email

• Phone

• Unique identifier

• Dates when transactions happened or will happen (new sale, renewal, and so on)

These fields are visually depicted in Figure 17.5.

The organization’s marketing business unit has the database fields and formats shown in Figure 17.6 (which compares the marketing data fields and formats with the sales data fields and formats).

As you can see, the two databases have fields that differ not only in their names but also in their formats and expected information. This is a classic case of an organization maintaining multiple databases without any uniformity. This lack of uniformity leads to formatting issues and errors due to overlapping or absent data. Data standardization has a number of benefits:

• Creating a single organizationwide view of all data fields

• Enhancing productivity and reducing costs due to data overlapping or errors

• Enabling the organization to maintain a clean and trusted master database that can be governed

• Enabling the same data to be leveraged across the organization

Data standardization is commonly done when onboarding datasets from internal or external databases that are based on varying definitions for fields and/or formats and transforming them into a trustworthy central dataset with common fields and a common data format.

Data standardization can be performed based on predefined business standards using rules or by leveraging third-party tools. While some data standardization might be trivial (such as capitalization of all characters, removing punctuation, or reordering date or time units), some datasets might require creation of rules or algorithms offered by third-party solutions, such as DataLadder, Datamation, and Experian.

Data Dictionary

A data dictionary is a centralized store for metadata (which, you’ll recall, is data about the data). It is not unusual to have very complex database structures and multiple fields across these databases. In such cases, the data dictionary is important because it can explain and expand on information such as the following:

• The names of fields in the databases

• The data types that are stored in those fields

• The roles that have access to the data in these databases

• The relationships between fields across the databases

• The security constraints that are applicable

Hence, data dictionaries can make it simpler to navigate through the tons of data that get processed from multiple sources in MDM.

Figure 17.7 shows a data dictionary developed for sales, marketing, and other sales-focused databases in an organization that sells products and associated services.

Figure 17.8 shows a database that is built leveraging this data dictionary.

There are two ways to build data dictionaries, and each method builds a different type of data dictionary:

• Active data dictionary: A data dictionary can be built automatically by a database (or databases) such that the data dictionary and the database(s) remain in sync. This is a major advantage as there is no need to keep the data dictionary updated manually.

• Passive data dictionary: A passive data dictionary is manually built and maintained. It is not referenced to any specific database and contains only reference information across one or more databases. Compared to an active data dictionary, a passive data dictionary requires more maintenance.

The key advantages of developing and maintaining data dictionaries are as follows:

• An expanded data dictionary allows for enhanced decision making based on analysis of better-understood information.

• A data dictionary promotes standardization of data and consistency across multiple domains in an organization.

• A data dictionary provides better documentation about data aspects (metadata).

Mergers and Acquisitions

This section sheds light on the topic of mergers and acquisitions and how MDM helps with governance of data and bringing together disparate data sources.

Organizations aim to grow their business reach and their customer base, and this growth can be organic growth (that is, growth via innovation or developing new products or services) or can occur via mergers and acquisitions (M&A), which pertains to merging with or acquiring another organization that can complement the product or service capabilities and therefore help increase the market size. In the case of M&A, an organization may have the right processes to ensure that there’s still a single source of truth after M&A is complete, or it may have to go through the rather painstaking process of finding the right datasets.

For example, consider an example. Say that Organization A develops CRM software and has a good market share. Customers are now expecting new features and an integrated analytics functions, so Organization A looks at acquiring a well-known analytics platform provider, Organization B. This acquisition will give Organization A access to a new client base—and that’s huge. However, after the acquisition, Organization A struggles to formulate a strategy to combine the customer data from Organization B and loses steam.

In this example, MDM would mitigate the problem by integrating with new data sources and creating a single master source that provides a single source of truth for all enterprise-related data, resulting in minimal errors and minimal redundancy in business processes and giving the outcomes expected. The key aspects from an M&A perspective are timely integration and utilization of data. In our example, if Organization A deployed and prepared the MDM solution before the acquisition, it would take a much shorter time for Organization A to realize the benefits and reach the outcomes.

MDM helps integrate data with varying attributes and formats from disparate sources of data; it offers a single unified representation of all information rather than leaving the information in silos. This integration can be achieved by merging data from multiple systems and managing the master data via MDM to improve not just the data usability but also data governance and streamlined data access, as discussed in the following sections.

Compliance with Policies and Regulations

MDM provides unification of data, which directly relates to how an organization can direct its people, processes, and technology to comply with organization policies as well as local and other regulations—with ease. The key aspect is that properly managed and unified data is reliable and trustworthy; therefore, the compliance with regulations is more seamless.

Where there is a unified view of all the organization’s information, regulated data can be viewed the same way across the whole organization. Thanks to MDM, the organization can more easily determine which data can or cannot be disclosed, which data requires extra security, which data needs access restrictions, and which data can or cannot be shared with any external parties.

Further, MDM platforms make it easier to adhere to regulation frameworks (for example, HIPAA), which means an organization doesn’t have to work it out on its own and go about doing many tasks to secure data. For example, a financial organization could leverage MDM to record, store, and submit know your customer (KYC) data to regulators as required.

Streamline Data Access

MDM brings master data to the right audience by streamlining data access. It streamlines access to data insights from MDM golden records as well as access to compliance data for regulators.

Following are some of the key ways that MDM impacts customer interactions:

• MDM offers a cohesive view of customers across the organization, which enables sales, services, marketing, and so on to approach customers effectively.

• MDM improves customer engagement as everyone has the same level of visibility into customer accounts.

• MDM reduces costs for reengaging with customers as a single campaign covers the desired customers without wasting money on duplicate efforts.

• MDM offers access to information across applications and enables automation for repetitive processes.

• MDM offers compliance insights, which would be difficult to obtain on a system-by-system basis.

• MDM improves decision making as updated data is available to drive insights.