Neiman Marcus, a major retailer, was also hit hard. Hackers floated internally unnoticed for eight months. The estimated number of customers exposed was initially 1.1 million. Later that estimate was reduced to 350,000. Malware was directly installed on terminals similar to the intrusion at Target.

Holden Security, based out of Milwaukee, claimed that a Russian hacker group stole 1.2 billion usernames and passwords. While it is believed that the hackers plan to use the information to send out email health-products spam, the full extent of the theft is not yet clear. Spamming our mailboxes may not seem like it has a large impact. If the hackers sell the lists to companies, you may receive email that must be manually deleted. Spam filters are not perfect. If you spend 1–2 s to identify a spam email that passed through the filters and push the delete button, possibly over 600,000 h of time has been wasted across people just like you. That is equivalent to an organization of over 300 people working an entire year doing nothing but deleting email.

Facebook claims over 1 billion active users and is one of the largest social networking sites in the world. It has faced constant criticism over its privacy policies. Several lawsuits and complaints to the Fair Trade Commission (FTC) have highlighted that Facebook’s policies allow the routine use of members’ images and names in commercial advertising without consent. Facebook has updated their Statement of Rights and Responsibilities and Data Use policies. It was later disclosed that Facebook ran experiments on their members in order to test the impact of user-interface changes to their members’ emotions and actions without consent. Facebook noted that recent updates to their data use policy clarified the language in the policy, but they did not actually change the policy that allows them to perform these experiments.

After a lengthy lawsuit in the European Union, Google launched the “right to be forgotten webform” to handle removal requests. European citizens can request that links to information about them be removed from search results. The webform is the first step toward supporting the court ruling. However, Google says the form is used to evaluate the request and the process of how to comply with the ruling and requests will come later. Information such as the needs for public information, timeliness of the information, and other factors will be considered in the request. Other search engines such as Bing or Yahoo must follow suit.

All of these stories highlight issues that affect you directly:

When any organization has a direct sales channel to consumers, they should also become focused on ensuring the customer experience is optimally tuned. The customer experience is often directly tied to product quality and supply chain variability. For example, if a manufacturer delivers a laptop to a customer, it may care about the following types of metrics:

These customer experience metrics reflect what customers have said impact their economic behavior over time. However, similar to all customer experience metrics, creating these metrics, setting targets, and monitoring them requires substantial amounts of integrated data from many different groups. Identifying and defining these metrics may take a substantial amount of time and may involve subcalculations with many nuances.

Here are some examples of the types of issues that may occur when defining these metrics:

As the metrics need to be sliced and diced, the number of combinations between the various factors such as hierarchies and master data becomes large (Fig. 1.1). Merely trying to establish a common vocabulary, a first step, can become a huge hurtle.

Here are some key thoughts to extract from this story:

As drugs, terrorism, and human trafficking have risen to the forefront of global politics, new policies have been directed at squeezing the money supply of these acts. Some estimate that the total amount of “bad” money flowing in the global economy to be close to 1 trillion US dollars annually. If “bad” money were a country it would make the top three in total Gross Domestic Product (GDP).

A money launderer takes money produced from illegal activities and cleans the money so that it can be used to purchase legitimate assets such as a house or car. Today, even clean money is a problem. Some donors send their clean money to bad people. For example, clean money, such as revenue generated from selling cigarettes, can turn dirty if the cigarettes were smuggled into a high-cigarette-tax states to generate higher profits. Sometimes, clean money is directly wired (electronic transfer) to terrorists. The entire money movement process is global, complex, and a source of concern for many countries. Ultimately, money in the hands of “bad” guys comes back to haunt sovereign states.

The Bank Secrecy Act of 1970 as well the USA Patriot Act of 2001 created authority and policies to reduce financial institutions’ role in funding terrorist activities. As part of the broad antimoney laundering agenda, multiple US government agencies, working in concert with other countries, enacted laws and imposed regulations to support antimoney laundering detection and prosecution.

Federal enforcement actions against banks have increased. Banks have been hit with Office of the Comptroller (OCC) consent orders, Financial Crimes Enforcement Network (FinCEN), or federal and state regulators for material weaknesses in their Anti-Money Laundering (AML) and Bank Secrecy Act (BSA) or The Currency and Foreign Transactions Reporting Act of 1970 programs. OCC consent orders must be resolved or the bank will be shut down or severely restricted. The list below shows some recent fines as documented on the FinCEN website:

Recently, regulators have started aggressively pursuing both corporate penalties as well as personal penalties against company officers in order to create an environment of deterrence.

There are specific regulatory requirements to ensure that a bank is not doing business with a money launderer. Financial institutions must perform checks, which are called “controls.”

For example, when a bank opens an account, it must perform Customer Due Diligence (CDD) to determine if the account is legit. If the due diligence results are negative, a bank can and should decline to open an account and even close an active account. CDD is part of a broader Know Your Customer (KYC) theme. KYC involves gathering significant amounts of information about a customer and using the information to assess customer risk.

Another example of a control is the set of steps taken before an electronic wire transfer can be sent. The receiver’s name, country, and other information must be validated. When a wire is created, the name and destination country are listed on the wire instructions. It should be easy to match the name and destination against a known sanctions list, but some banks use multiple lists internally to represent country codes and these lists are sometimes conflicting. Various international groups such as the International Standards Organization (ISO) have created “master” lists of frequently used country codes. However, the ISO list is not always quickly updated with changes, and it is possible to have country names appear or change and not be reflected in the ISO list. Furthermore, the two- and three-letter codes used to represent different countries can be the same code on different lists but map to different countries on an ISO and a non-ISO list.

Another control is used when assessing commercial banking services. Similar to the due diligence with an individual, a bank wishing to do business with a company must understand who that company is, how it generates money, and assess its AML risk. For example, a company that is a money services business or a “cash” oriented business like a cleaning or massage service will have a higher AML risk score. Validating a company’s “industry affiliation” is an important step in the process. Like country names, industry affiliations are kept on standardized lists and nonstandardized lists. Similar to the country name and abbreviation issue, industry affiliation can be misunderstood. FinCEN recently assessed a civil money penalty on Mian, doing business as Tower Package Store, because it determined, with help of the Internal Revenue Services (IRS), that Mian was performing as a money services business. A favorable industry affiliation can sometimes dramatically reduce the risk score in some risk models and create a sense of false security.

In addition to controls, simply defining the terms used in the policies can be difficult. In 2012, the FinCEN was preparing to issue updated guidelines on its policies and per its policy solicited feedback on the changes before finalizing the guidelines. The policy update was fairly explicit, but there were areas that needed further clarity. In their public feedback, the American Banker’s Association (ABA) expressed that more definition was needed:

Here are some key thoughts to extract from these stories:

Health information is private information. A broad set of policies has been enacted in the healthcare industry to enhance protection of personal healthcare information (PHI). Title II of the Healthcare Insurance Portability and Accountability Act (HIPAA) established national standards for electronic healthcare transactions and national identifiers for providers, health insurance plans, and employers. It also established civil money penalties for violations of its provisions.

The healthcare industry, often led by the federal government, create initiatives to accelerate value delivery with special emphasis on high-cost chronic diseases. For example, National Institutes of Health (NIH) Department from the U.S. Department of Health and Human Services sponsored the cancer Biomedical Informatics Grid (caBIG) project, now retired. Focused on improving cancer outcomes, the program accelerated investment and capabilities along several fronts. One focus area was on improving access and use of cancer data including extensive amounts of PHI. It has been and continues to be a challenge to share data in healthcare settings due to HIPAA and other policies. Because cancer funding is highly fragmented spread across many constituent states, cancer-related data is created and housed in many different research, government, and commercial locations. To make this data more sharable, the caBIG project worked to make data dictionaries, medical taxonomies, and the ability to run queries across distributed databases as easy as using a spreadsheet. This makes data more accessible and usable to all cancer researchers.

The caBIG program funded projects to establish common vocabularies. Common vocabularies help researchers identify data for their queries. A word describing a condition used in one location needs to mean the same thing in another location. Medicine is still largely implemented and practiced locally with localized standards and practices. Merely identifying data of interest to a particular research query is the first step. There are many cultural and policy impediments to moving and using healthcare data in consolidated facilities. These impediments reduce the ability to run integrated analysis across larger datasets to improve analysis results. Additional caBIG-funded projects created tools to allow a cancer query to execute on a local database and return results to the query requester. The highly fragmented state of healthcare research databases coupled with privacy restrictions has led to greatly reduced access to data for analysis and significantly higher healthcare costs.

Understanding what data is available and its fitness for use is an important step in analyzing data to answer questions. The Center for Medicare Services (CMS) was ordered to provide broader access to the data it collects as part of its mission of administering Medicare. CMS made a dataset available that describes reimbursement dollars and prescription counts for physicians that provide Medicare services.

Once the data was released, many articles were published by experts to explain the data. For example, it was found that some physicians had prescribed more expensive drugs even though proven, lower-cost alternatives were readily available. The data also suggested that some physicians wrote excessive prescriptions counts, high enough that it was nearly impossible to correlate the counts with any form of patient schedule. It was later explained that prescriptions are often attributed to a department head or another physician manager and not the physician providing the actual care.

These datasets were also truncated. Due to HIPAA concerns, physicians who had fewer patients than a threshold were excluded because it is possible to infer from very small datasets actual patient specifics.

The cost of combining data together to answer healthcare queries can be high. As part of the desire to improve clinical analysis, the healthcare industry decided to migrate from a specific set of industry classification codes to a newer set. The industry is in the process of moving to the ICD-10 standard from the current ICD-9 standard. The ICD codes describe diagnosis and procedures performed while delivering healthcare services. The diagnosis codes are the boxes on the form your healthcare physician fills out during a doctor’s visit. ICD-10 codes are more descriptive than ICD-9 and allow analysts to perform more fine-grained analysis between the diagnosis, the procedures used to treat the underlying issue, and eventually outcomes. ICD-10 also enhances the ability to analyze patient data by episode of care (longitudinal analysis) versus one-off interactions with the healthcare industry. Even though it may seem like a fairly straightforward process to migrate from ICD-9 to ICD-10, the road has been very difficult. The industry started moving to ICD-10 a decade ago. Fig. 1.2 summarizes how ICD-9 and ICD-10 differ.

In addition to the cost of migrating technical systems being much larger than expected it has also been found that changing diagnosis codes changes reimbursement levels because the reimbursement rules are based on these codes. Changes affecting reimbursement are highly susceptible to political forces and conservative change. As you might expect, there have been multiple delays in implementation deadlines, increased costs as well as extensive and advanced financial analysis and modeling around the changes. Healthcare organizations simulate code changes based on their current book of business (​it refers to the current customers and transactions they are experiencing) and project the impact to future reimbursement under the new codes. Based on the simulation, they provide feedback to the government. Then the cycle repeats. If reimbursement goes down, negotiations around reimbursement become intertwined with the technical changes, causing further delays and costs.

Here are some key thoughts to extract from these stories:

Information management grew into a strong discipline over two decades ago. There are differences between “information management” and “data management,” but the distinction between the two terms is not important for this book and we will use the two terms interchangeably.

There has been rapid convergence to best practices for managing and deploying information management resources and capabilities. However, consistently achieving expected business outcomes in the information management areas has remained elusive. Reasons include:

The top three issues above are really caused by management deficiencies: starting projects when they should not be started or when they lack adequate support to be successful.

We have seen severe deficiencies in many organizations’ capability to define realistic roles for the IT group. Business groups sometimes contribute to this problem by completely delegating oversight and responsibility of information management projects to the IT group. IT groups are often eager to absorb responsibility for creating full solutions. However, business groups do not always need end-to-end solutions as they can cost effectively do part of the work. Sometimes, having IT do less and ensuring responsibility is placed where it is best executed is a better solution. Organizations that experience multiple, successive information management project failures often have IT groups carrying too much scope. Business groups that have their own analytics applications, such as SAS or R, are fully capable of working on the “last mile” of information delivery.

It’s better to stick to an “understand the jobs” approach. If we understand the jobs that need to be performed, we know what needs to be done regardless of who does the job. Once the organization’s dynamics have been mapped into the “jobs,” the assignments for who performs which jobs can be assigned and the Playbook adapted appropriately.

The Playbook provides workstreams across stages and phases that collectively perform the data governance jobs. The Playbook workstreams we cover in this book intersect the following information management areas:

The term can be confusing without more context. Business resources often use the term to refer to activity that uses data and math to answer business questions. IT resources often use the term to refer to reporting and reporting applications.

Using any of the above definitions, business intelligence depends on data. The data used in business intelligence is often:

Business intelligence data is a major beneficiary of data governance activities. Data suitable for aggregation and integration must be cleansed to a certain degree. The field of data cleansing and the concept of cleansing data can be a bit abstract, but we define it as the activity that changes data so it is more suitable for use. Restructuring dates consistently so that sales figures can be aggregated is an example of data cleansing.

Integrating or aggregating data often occurs across functional groups. Data governance can help ensure that each data concept can be conceptually “added” together. Data governance is not a “specific” answer to solve disconnects between groups that do not want to agree to integrate and aggregate their data. If groups do not agree, they do not agree. While the Playbook has processes that can be used across groups, organizational dynamics can still affect the ability to execute the process regardless of who owns the data governance jobs. We provide some insight into how to manage these organizational dynamics in later chapters.

Attestable data is data that has data controls applied to it. Data controls allow the data consumer to attest that the information is correct for their needs. Attestable data requires structured processes and well-defined controls and the Playbook provides activities for creating these controls.

The best way to think about the Playbook and business intelligence is that the Playbook is not business intelligence – the Playbook makes business intelligence better.

Analytics was typically physics- and engineering-oriented rather than business-oriented. Applied and theoretical mathematics were targeted at understanding and explaining various physical phenomena. For example, new mathematical approaches have been developed to understand why Newtonian physics break down at small scale.

As populations grew, more scientists and engineers were available to work on other problems. Mathematics turned its attention to solving business and social issues such as improving health, alleviating traffic, or building timesaving machines such as dishwashers and airplanes. Mathematics and statistics became a potent corporate weapon, but initially only large corporations could afford the talent.

At some point, the use of mathematics went into hyperdrive. Starting with mundane tasks such as reducing the costs of mailings for magazine subscription solicitations (ie, junk mail), the field of statistics morphed into the field of data mining and machine learning. Thousands of analytically oriented employees worked on problems such as how to optimally select ads and optimally place them on websites. Fairly quickly, advertising became the primary focus of the largest companies on Earth.

The widespread availability of talent and infrastructure led to dramatic decreases in the cost of deploying business analytics, and a funny thing happened on the way to profit heaven. The data used in the analytics suddenly became a bottleneck in achieving better analytical outcomes. Data is noisy due to the nature of data collection (eg, web page data entry) and is often conflicting. As data was integrated from significantly larger numbers of disparate sources, data quality reduced the effectiveness of the analytical models.

Large areas of the information management space can trace its growth to the rise of analytics and the rise of systems to create analytical outputs. New industries arose to address the data warehousing space. A plethora of data-quality tools, now integral to the dataflow processing inside a company, became routine.

The rise of data-quality issues and their impact on analytics rapidly created the need to drive agreement across aggregation, integration, cleansing, and attestable data. New issues arose such as whom in an organization should have access to certain data elements? Who gets to make that decision? Although these questions are really business management questions, employees began to believe that data governance should answer these questions.

This book is not about the math behind statistics or data mining, but it does deal with the need for improved data and is specifically designed to make consuming data in analytical applications consistent for analysts. Just like business intelligence, the Playbook makes data mining better.

With the rise of Big Data, NoSQL databases also took center stage. No longer restricted to a single machine, NoSQL databases promise to span commodity infrastructure on-demand to solve large data management problems. Many believe that NoSQL databases have a flexible schema more suitable to applications that require highly connected data and specific access patterns. With different guarantees than traditional relational database management systems (RDBS), NoSQL databases and related technologies such as graph databases have become a common complement to Big Data.

For example, many NoSQL databases running on clusters offer “eventual consistency.” Eventual consistency fits social media interactions well. Most social media users are fine if the time it takes to reflect a new “post” is a few minutes or hours. The same guarantees applied in the business world, say around trading data, might be costly. The use of Big Data for storing specific types of data across large clusters provides cost-effective solutions to web-dominant applications that require enormous scale or to business applications where these guarantees are sufficient.

Big Data technologies became synonymous with cloud computing. Because Big Data technologies often use parallel computation and data management processes, they are applications that inherently assume that single-system limitations will always be present. These technologies assume large clusters. Early Web companies relied on commodity clusters to scale as their businesses scaled upward, ie, scaling horizontally instead of vertically became the norm. Public clouds, and even private clouds to some extent, allowed these companies to take advantage of pay-as-you-go plans as their needs increased. As security and other factors became more significant to some commercial clients, public clouds and private clouds became the basis for deploying Big Data technologies.

The motivation behind Big Data technologies is rooted in a specific set of processing scenarios. Today, these technologies are entering mainstream use and are used in other scenarios such as offloading processing from more expensive, dedicated systems such as datawarehouse appliances. Big Data and NoSQL have shifted the cost and capability landscape and will continue to do so as these technologies mature. Big Data enjoys the most success where the technology fits the need. For example, Big Data is very useful in the life sciences upstream in the discovery process where large datasets generated from assays generate enormous datasets and eventual consistency guarantees are fine.

Interestingly, current parallel extract, transform, and load (ETL) tools already contain the same components that popular Hadoop-based Big Data systems are developing such as parallel filesystems, resource management, and scheduling and parallel algorithms. However, the cost of purchasing and keeping the talent needed to run ETL tools can be large and beyond the reach of many smaller companies. Whether true or not, some businesses feel that the cost curve for newer Big Data technologies has shifted enough to enable these technologies at their organization. While Big Data technology will certainly be misapplied to situations that do not benefit from its value proposition, Big Data will be present in the landscape and data governance needs to take them into account.

Big Data technologies have the capabilities needed to store and process data. While the specific technologies may be different compared to traditional data management technologies, Big Data and traditional technologies are conceptually the same – they are applications that hold, manage, and process data.

Big Data deployments can strain a data governance program because of issues such as:

While it may seem that the integration of Big Data is really just a policy definition exercise, elements of the technology at different cost points makes the data volume and diversity managed under Big Data “bigger.” Data may move faster and be more complex. Many organizations already find it difficult to apply data governance to the data they oversee today in traditional technologies. Data governance needs to eventually have 100% coverage over data in the organization. Big Data deployments increase the surface area that must be covered. These issues are not insurmountable. Just as vendor products make using Big Data technologies easier, the data managed under Big Data technologies will be added to the queue of data governance work. When it comes to Big Data, the Playbook applies just as it does to any other data.

The maturity levels are often set by an independent body or based on analysis of the industry or function. In addition to serving as a great communication and consensus-driving tool about current capabilities, the categories and levels also help educate an organization on what capabilities need improvement. Most maturity models are descendants of the Carnegie Mellon University’s Software Engineering Institute’s Capability Maturity Model Integration (CMMI) for application development.

There are multiple maturity models available in the information management space. Many of the information management maturity models such as the Enterprise Data Management Council’s Data Management Maturity Model (http://www.edmcouncil.org/dmm) overlap somewhat with the capabilities needed to implement data governance well. CMMI Institute’s own Data Management Maturity Model (DMM) is specifically designed to align your data management strategy to business results. These organizations also work together to help improve alignment between the models. We greatly encourage you to use these models to understand the capabilities that may be important to you across the entire data management landscape.

The Playbook is not a capability maturity model. While we find that maturity models are great at communicating and identifying improvement areas, organizations are constrained because they do not have a detailed execution plan for data governance that provides day-to-day guidance. The Playbook describes an execution path represented by a series of specific activities that should be performed to successfully perform the data governance jobs. The areas described in this book cover those that are most commonly addressed by many organizations.

There are variations in how some data governance activities can be performed. You also need to look into your organization’s toolbox to find and select tools that help you execute. The selected set of options forms the “toolkit,” which could be very simple, such as using spreadsheets and email, or it could be more complex. For example, using advanced business glossary and workflows tools to manage the workload and achieve your complex communication and validation requirements is an advanced approach.

The Playbook contains a basic set of activities needed to accomplish the jobs in a reasonable, cost-effective manner. Essentially it describes how to conduct data governance operations. The Playbook is designed to be customized, and optional activities added by an organization can reflect different maturity levels.

We tend to think about maturity along two distinct dimensions:

Fig. 1.3 shows that maturity along these two dimensions can vary not only company to company but also within a company, department by department. Typically, maturity starts low. Different groups often wind up in different parts of the grid over time. For example, as Human Resources become more quantitative, its potential to move has changed allowing it to mature its execution.

These two maturity dimensions are the “breadth” and “depth” of your Playbook. Execution maturity increases if you adopt a Playbook and adapt it to your organization. Service maturity increases when we discuss data governance as an operational process. We address Execution maturity by providing a process to adapt the Playbook to your organization and update it over time. Service maturity increases over time as you repeatedly apply the Playbook to your organization’s data. Most maturity model literature calls out consistency of execution as a key outcome of using a maturity model.