The first area that leverages policy-based executive mandates is assessment and planning, the focus of this chapter. We break down and describe in detail each of the assessment targets in the following two pages. Here we just want to highlight the assessment approach we’re using to look at the conditions, controls, and capabilities for each of those subjects and the activities that should be provided for teams to perform. This assessment will provide us with the scope of data and the scope of the people necessary to close the gaps related to issues with data and issues with our teams’ capabilities. Adding to a team’s experience while improving data conditions and controls is a key construct in the Playbook. The output from the assessment is a series of Playbook activities that are required to fill the identified gaps. Those activities are then sequenced into an execution sequence, which can be added to the overall roadmap you are using for your program or permanent function.

The second area is in the execution of Playbook activities and its operational aspects. When you have a large program or corporate function for data and analytic governance, build out a roadmap that identifies which areas of the business, subjects of data, and terms you want to expand and improve upon. Identifying and expanding your Playbook set of activities to do this is a key step, because it provides standard methods and measures for your execution. This is typically a requirement of solid policies and procedures. Many of the activities are actually operational governance practices and can operate as an ongoing service. Identifying such activities is a compass in the Playbook, and many firms expand this area of the Playbook to describe large, complex global data services established at the request of the business or in partnership with them. Some of these services transition key resources during data administration and work in lines of business over to a global data service. Data services, like any other enterprise service, have service-level agreements and metrics associated with their delivery and outcomes. Data services often save the business time and money around their development and ongoing operational activities. A data service example is the duplication and validation of names and addresses against live and dead name indexes to ensure we reduce the scrap and rework of returned mail. More complex examples come from the Big Data arena, where we see new delivery tracking and emerging drone and automated delivery methods driving the need for much more precise delivery information. Identifying the right measures, and the best way to communicate those measures as they change and over periodic requirements, is the last stage in the execution steps.

Once the measures have been defined, they need to be vetted against the executive mandate to ensure all of the aspects of the policy-driven mandate are being addressed with regular measures and impact monitoring. This very dynamic area requires consistent measures as well as some flexible measures that can change over time. Flexible measures refer to services that change in scale, nature, or scope over time. Standard measures around data quality, availability, and impacts are maintained with baselines. Measures related to execution and resource levels are maintained against baselines over time. Executives can expect to see comparisons of long-term resources allocated after levels are expended and outcomes achieved.

The Playbook assessment process is focused on three key areas, which collectively provide a clear view of the current conditions, controls, and capabilities of the enterprise, with regard to critical data and analytics. Many quality and governance programs focus on the conditions, and to some extent the capabilities, but often lack permanent controls and the measures that prove their efficacy. Balancing conditions, controls, and capabilities, as the Playbook suggests, ensures that our execution and operational support levels are consistently high enough to certify integrity and quality in our critical data and analytics. It’s also worth noting that this approach focuses on assessing the current state of these areas, in order to create baseline measures that support ongoing reporting and heat mapping. In addition, the outcomes of the assessment work are inputs for prioritization and planning along with the demand management cycle, which we will address later. In order to make the assessment effective in terms of measuring the gaps, it’s important to have baseline measures for each of the three areas in mind before you start. In this way, you ensure that your assessment is a fit-gap assessment that produces a list of gaps from expected values. Many data governance and related data management maturity models and assessment tools have been developed and published in the last 10 years. Each of these has unique value, but we came to the conclusion that the Playbook assessment approach was essential. This is because our emphasis includes a balanced focus on controls, whereas others are not as clear about that critical area. For example, it is important to be able to assess your capabilities with the controls focus in mind. Having strong data stewardship or analytic quality skill sets does not always translate into solid, reliable, and efficient controls in those areas. Thus it’s important to look at the controls as well as the capabilities in order to gauge the amount of energy that is being consistently applied to operating solid controls. It is also important to remember that, as capabilities improve, controls efficiency should improve, because it takes less effort from a well-schooled and experienced resource to identify, put in place, and operate an effective control. So we can expect to see improvements across these three areas as they occur within the three areas. Let’s look at the each of the areas to understand more about what we assess within.

The Current Conditions area focuses on data stewardship, governance, and quality management. It also includes the cataloging and operation of business analytics. Assessing the current condition requires at least two elements. These elements have been detailed in very mature programs and corporate functions that are based on the notion of “trust and verify.” Starting with assessment surveys, while communicating an ongoing review of the work being done, enforces accountability in your stewards. Asking a data steward if they keep and maintain the catalog of their critical data is useful. But asking those questions while simultaneously reviewing that catalog and discussing its gaps and omissions is far more effective in assessing the situation and, it communicates the level of accountability you expect from stewards going forward. Similarly, asking about the business analytics, change-control process, while simultaneously reviewing examples of currently used production analytics, is far more effective in validating the level of control in place and communicating the expectations for controls going forward. Finally, it’s worth noting that you will amass a series of worked examples that can be used for training and orientation of new stewards. Additionally, internal and external audit and regulatory experts are always looking for evidence of controls sufficiency in the conditions of data and analytics. They know that testing controls is an important part of their job, and that testing the way those controls result in solid data and analytics proves that the controls are being used consistently.

Assessing current controls is often best done in conjunction with internal audit and risk management professionals. At minimum, engaging these professionals in defining controls objectives and sufficiency tests is far more likely to result in a control’s assessment profile that meets their requirements as well as your own. Risk and audit professionals are also continuously monitoring industry and public threats, risks, and best practices. This makes them solid partners in defining controls, based on the risk and exposure we face with data and analytics. Assessing data and analytic controls often requires some segregation of business-based audit and balance controls, from the IT-based technical controls. We are seeking an assessment of control effectiveness and efficiency, so we must test those who operate the controls in order to understand how well the controls themselves are being applied. We are simultaneously testing for whether the controls actually help us monitor, manage, and remediate risk and exposure based on our specific risk profile. We don’t need to control or test controls for risks that we do not face or that do not rise to a level that necessitates controls. We do need to ensure that appropriate controls are in place and tested where there is known risk and exposure. Finally, it’s worth noting that assessments should be conducted on a periodic basis to create a baseline for the ongoing controls’ effectiveness and cost. We should also be collecting anecdotal information about controls that are known to have prevented loss or other risk from being realized.

Due to the way we interact with certain people, it can be difficult to measure people’s data and analytics capability without skewing the measures. There are many complications identified with measuring people’s behavior and attitudes in business settings. In this case, measuring capabilities is best quantified through their collective experience in executing the Playbook, managing controls, and reporting on status and issues. Our experience indicates that tracking people’s outcomes with data and analytics governance is the surest way to determine their overall capability levels and commitment to quality. Here, as in current conditions, we’re looking across data stewardship, governance, quality management, and analytics to understand how experienced and effective our people are. People’s experience and the effectiveness of their part of our program or corporate function can be combined to determine our capability level in that area. However, we must not become too monolithic in the expression of summary information. Many capability models find it very difficult to provide granular views of differing capability levels or maturity levels across lines of business, geographies, and other business segments. We find it most useful to survey and observe, particularly with newer, less experienced people. Many times we see new people come into a company with a very long resume and list of accomplishments, but these do not always apply directly to the challenges in their new environment. So tracking their activities through survey and observation can identify where they are not aligning with the company’s approach. Similarly, when you have somebody who is relatively inexperienced but rapidly adapts to challenges, this sort of assessment approach can highlight those abilities.

Now that we’ve covered the basic areas to assess and talked about the use of surveying and observation, we can address another input in the assessment process. Status reports compiled by the overall program, which describe the progress of their work across both the organization and the data scope, are a primary input. From quarter-to-quarter and year-to-year, we can expect to see progress in expanding a program or function across the company and addressing commensurately larger bodies of data and analytics. Tracking that progression across the company and its data is a critical part of evaluating the growth of the program’s coverage. Many approaches use coverage as their primary measure of program success. We’ve found that doing so creates a gap between the perceived success rate and actual success rate, because what we identify as covered is often not treated in a consistent manner. We therefore use our assessment survey and observations to confirm that each area of the firm and subject area of data formally addressed by the programmer function is actually completely addressed to the level identified. Trusting in openly verifying with your business partners, who have invested time and energy into doing this well, is the surest way to confirm the coverage leads to success.

All of the assessment outputs should be considered as evidence of progress and improvements. Where there are clear gaps in the current conditions, controls, and capabilities, we can circle back to the Playbook to identify the required sequence of activities to close the gaps or to bring up our overall levels as needed. Please remember that the Playbook can be customized and extended to address things that are critical or somewhat distinct to your company. But these assessed gaps are only part of our input into the overall roadmap and work plan for Playbook-based activities across the company. Another key input is a demand management function. We address this in other areas of the book, but need to summarize some steps here to show how we add inputs from it into the overall roadmap and work-planning effort.

Solid demand management for data and analytics engages stakeholders to understand their problems, issues, requests and enhancement needs but does not stop there. These various requirements for improvements and services must be vetted against two more levels. The first is management data initiatives, which can broadly address data and analytics needs that multiple groups of the company or key functions of the company have identified as gaps in today’s environment. Those gaps often lead to a major project or initiative that addresses a number of the individual problems or issues we collected before. These initiatives also establish broader priorities and improve data and analytics based on what you choose to focus on and how far you go in making improvements. These initiatives are also critical because they have managed to acquire funding and executive sponsorship, and so are already a part of the committed demand pipeline.

The final level we have to include in our analysis of demand is a set of enterprise data and analytics priorities. These typically emerge from the executive team. Some of these will be reflected in the type of management data initiatives that are underway. Additional enterprise priorities will be reflected in strategic dashboards, goals and objectives, and mission maps, which the enterprise uses to drive organizational behavior. It’s critical to understand the enterprise data priorities in some sort of ranked order, and these must be tied to desired outcomes, goals, or measures in order to be useful. We typically weigh the strategic goals by associating the initiatives that we reviewed and the request that we have of them. This gives us the ability to prioritize the coverage of initiatives and actions for requests and problems against the enterprise stated goals.

The combination of all of these components enables us to develop and maintain a roadmap or transition plan for data and analytics. It’s important that this plan have a communicated sequence, along with the logic behind why things are sequenced the way the plan indicates. Therefore we need to point back to the enterprise data priorities and management data initiatives in order to indicate why we have sequenced our work in the manner the plan suggests. This plan produces deployment schedules, which become a work order for various parts of the organization to engage in Playbook activities. It also drives the supported activities and initiatives for the shared or global data services partners.

This demand management process adds a corporate management and local set of priorities to the assessment outputs, which helps us craft our overall pipeline and sequence a roadmap based on the priorities of the company. It allows us to adjust the prioritization we use from the demand management side against the gaps that the assessment has identified we need to fill. Demand management can also point out emerging areas where additional data and analytics capabilities are needed and may in fact need to be added to the Playbook. Recent examples include the ways companies are mapping Big Data with conventional database and other technologies. We will talk a bit about the data in a future chapter, but the first thing to think about in terms of governing is knowing when it makes sense to invest in the first place. Thus this approach allows us to balance the current assessments gaps and issues with the demand and priorities already in place, so that we address all of our issues in a prioritized way.