CHAPTER 7

Conclusion

Capturing the Value of Big Data Projects

It is about adding value.

This book has defined Big Data, described its important analytical characteristics, and illustrated some of the many ways managers and executives can utilize Big Data to support business operations and improve service delivery. The Big Data picture painted thus is quite complex. But simply understanding the Big Data picture is not the goal of an effective decision maker. Effective decision makers and executives are constantly and clearly focusing attention and concentration on the prize: applying the Big Data findings and obtaining strategic and tactical results by capturing the promise of value in the data and applying this to improve services. The methods used to extract understandable and usable information that provide organizations with the knowledge to make highly improved operational, tactical, and strategic decisions must be combined with both a sound understanding of methods with technical and methodological names, and clear business strategies shaping how the knowledge derived from the Big Data can be effectively employed.

The key for a manager is not in simply being able to describe the characteristics of these tools, or in wildly and randomly applying the tools to all the data owned or available to an organization. It is in deeply appreciating the business and decisional impacts of the characteristics of the Big Data. Two examples will show why the Big Data analysis may be so useful or even critical to success of a service, product, or business.

The reports and research documenting the value of Big Data have been arriving for several years. Bain research surveys of executives of more than 400 companies with more than a billion dollars of revenues have shown that companies that are good at applying analytics and changing their business due to analytic findings are twice as likely to be in the highest quarter of the financial performers in their industry, are three times more likely to implement the executive decision as they desire, and are five times as fast as making decisions (Wegener and Sinha 2013).

Examples of value can be very specific. The Bain research cites Nest as an example of a company that not only applies remote control thermostat technology to control the environment of a home through the Web, but additionally uses crowdsourced intelligence to determine when and how the home’s thermostats are set and changed. These data are then associated with other factors such as the home’s local environmental conditions, weather, physical location, and construction type to assist in determining the setting to create a more pleasing living environment inside the home (Wegener and Sinha 2013).

Where is the value being found by using Big Data and data analytics? Early leaders may be financial services, technology, and health care. Examples include mail-order pharmacies that found increases in service calls associated with specific refill windows for prescriptions. Deep analysis showed that customers had variable dosages. The pharmacy responded with periodic predictive calls inquiring how many pills customers had remaining and reduced time-consuming customer service calls and emergency refills (Wegener and Sinha 2013). Examples are beginning to proliferate—with call centers routing executive request, more important clients, or premium consumers based upon telephone numbers. In the airline industry, these data can be combined with flight status data that can possibly predict why one is calling (flight diversion or delay), and used to deliver a more rapid update without taking time to receive, record, analyze, and identify a response.

Where can value be expected to be realized? Look for Big Data to make information (of many types—customers, products, inventories, complaints, and so on) transparent and usable at much higher frequency. Digitized transactional data is more available and precise thus exposing relationships to outcome variables showing true costs of performance (product variability, personal absences, traffic blockages, etc.). Value is realized by acting on the enormous opportunities to experiment with clients, alternate deliveries, service variations, and product improvements. One can then compare success rates thereby making managers and executives into scientists who can realistically apply a more scientific or analytical method to seek out and pinpoint the key actions and reasons behind sales increase or decrease or behaviors finely tuned likes and dislikes of customers (Manyika et al. 2011). Value is thus found in multiple areas of businesses and in numerous processes—through the tailoring of services, managerial decision-making, and the rapid design and development of generations of products and services.

The more generic lists of possible potential business benefits include timely insights from the vast amounts of data; real-time monitoring and forecasting of actions and activities that effect performance; ability to interchange the data tools used such as SAP HANA, SAP Sybase^®, SAP Intelligence Analysis for Public Sector application by Palantir, Kapow^®, Hadoop; improvement through including data of different types and increasing velocity; and better validation and verification, better risk recognition, assessment, and decision making.

A few common characteristics of Big Data insights include the following.

The first example is one of swatting—where in current “practice” someone may report a murder, kidnapping, shooting, or some other crime that requires a swat team response. The team is then directed to an innocent’s location, where doors are broken in, and the “swat rescue” is imposed by an unknowing police but well-intentioned rescue force under the pressure of potentially life-saving consequences. What occurs is “swatting, ” a more and more prevalent Internet-aided trick on the police in which cybercriminals report a criminal hostage situation or shooter on the loose threat with the goal of unleashing a SWAT response on an unprepared and unknowing individual.

How does this relate to Big Data? First, careful analysis of the data may identify the perpetrators. But further analysis may enable one to examine and understand the linkages in the network, and how to utilize the connectivity and speed of the functional communication networks that “share” the information or report in real time. Swatting incidents do not even have to be real to illustrate how communications mapping can demonstrate the power of Big Data analysis. A recent viral Internet video depicts a 15-year-old boy being convicted of this crime and sentenced to a long jail term for his actions. Many believe the “conviction story” is true, but careful analysis of the data shows it is false. (No one was convicted of the crime—it did not even occur.) However, the Big Data will map the rapid dissemination and communication of this story. When this mapping is combined with survey and Big Data analysis, the power or reach of the story is clearly demonstrated.

A second similar example is of a non melting ice-cream sandwich. Several videos show that in 80° sunny temperatures, the ice-cream sandwich does not melt or change its shape. It is embarrassing, and the video obviously is intended to reflect negatively on this particular treat. The Big Data impact is derived from the transmission of this message, tracking of views, and viral nature of the communication generated. The communication paths themselves, wide dissemination of the “event” or message through channels, methods by which the messages are “picked-up” and passed, and immediacy with which the social media communicate the real or supposedly real event are useful for planning communication events and developing marketing programs. The networks of communication paths can be mapped, the impact can easily be seen in supporting social communication and video views, and then used to project an attached or implanted service message, product capability, or to defuse (or accentuate) a problem event or situation.

Finally, there is value in the data beyond that than can be captured within the processes of the organization. But there are no official guidelines for assessing data value because data is not a real asset like a factory or tangible cash. The traffic in information is large and generates revenue, but standard methods for valuing data must be developed. The issue is great. The Wall Street Journal (WSJ) reports that supermarket operator Kroger Co. has more than 2,600 stores where it tracks the purchases of 55 million loyalty-card members. Data are analyzed for trends and then, through a joint venture, sold to the store vendors. The consumer-products makers purchase this information and are able to thereby adjust products and marketing to match consumer choices, likes, and dislikes. It is estimated that Kroger receives $100 million a year from such data sales (Monga 2014).

Metrics: What Are They and How Should They Be Used?

Performance assessment is a critical foundation of any agreement, deal, or transactional relationship established between parties involved in a service arrangement. And managers, executives, and organizations are all judged by performance. The judgment process requires that measures or metrics be set that will aid in establishing the value of a service, and if the service is effective. The metric may further aid in setting behaviors and actions and in assessing tactics and strategies, thus determining managerial and executive decisions and actions. Metrics can be constructed to evaluate the “value” by determining degrees of performance and compliance, by encouraging future performance improvements, by demonstrating increases in effectiveness and efficiency, and by providing managers with mechanisms to determine the levels of effective internal controls.

Determining Value

Determining the value that can be or is derived from analysis of Big Data is a critical problem from many perspectives. The manager may realistically focus on quantitatively determining the impact of Big Data on the customer’s overall satisfaction (with a single purchase or service, or a reoccurring use), with the performance and productivity of the organization, and with the impact on the organization’s workforce. Questions that executives and managers must ask are the keys to assessing value. Do the data show that the outcomes are in full agreement with the mission and goals of the organization? Is the product or service less costly to produce, or of higher quality? Does the service meet all (or more) of the needs and requirements of a customer? Does the service exhibit higher quality characteristics? Are inventory reduced or are inventory turns increased? Are corporate promises being met (explicit or implied)? Are services being delivered in a more well-timed frame (thus reducing backlogs or shortening wait times for customers) and increasing the number of customer positive communications to others?

Measurement Costs of Big Data

Of course, there are other organization investment metrics that can be assessed that will also demand that costs of producing the benefit be fully assessed. This calculation requires that one begin measuring the value delivered to businesses from Big Data and its corresponding analysis by defining the expenses associated with producing analytics, and including the infrastructure components required to structure, cleanse, analyze, and present the Big Data to decision makers. The costs are openly assessed in their initial collection. They are composed of the costs of tools (software, computation, management, and storage), costs to acquire data (either by purchase or direct collection), analysis costs (personnel), and visualization costs for preparing the data for consumption by managers and executives and to associate, support, or relate the data to the decisions to be made.

Using Organization Performance Metrics to Explain Big Data Findings

Assessing Big Data’s value is a far more difficult problem when the analysis must deliver explanations as to why end measures of organizational performance previously identified are being obtained. The Big Data and analytic results must be correlated with increased sales, customer growth, increased size of purchases, profits, faster inventory turns, and the list goes on and on. The explanatory process requires that performance metrics for the organization be developed by involving the employees who are directly accountable for the effort to be assessed. They are knowledgeable about the processes and activities involved in the work. The steps in the value assessment process include:

• Establishing the essential work activities and customer requirements
• Aligning work outcomes to customer requirements
• Setting measures for the critical work processes or critical results
• Establishing work goals, criterions, or benchmarks

Goals may be set at multiple levels, for end objectives related to the mission or function, direct targets that match divisions of responsibility within primary functions, and end metrics designed to link to specific improvements characterizing advances for each criteria.

These must be quantifiable and concrete targets based on individual expected work results.

With careful organizational planning, the measures themselves will be clear and focused, not subject to misunderstanding, quantified and compared to other data through statistical analysis, reasonable and credible, and collected under normalized or expected conditions. Measures must also appropriately fall within the organization’s constraints and be actionable within a time determined frame. Measures may take various forms—and be seen as trends or rate-of-change (over time) data including views set against standards and benchmarks—or be fixed with baselines to be established at some future point, or determined incrementally via milestones.

A question still remains—when are the organizational metrics good? Are they useful or important for the organization’s success? Although they may be done well, following all appropriate applied rules and procedures, a manager still wonders—is the analytical data good? There are no silver bullets guaranteeing how the quality of the metrics can be assured. Answering key questions can help to make this determination. The straightforward questions listed below will point the way.

First, ask oneself about the content: Was it objectively measured? Was there a clear statement of the goal or result? Did the metric support customer service requirements and meet all compliance requirements? Did it focus on effectiveness and efficiency of the system being measured? Secondly, ask about the properties: Does the metric allow for meaningful trend or statistical analysis, meet any industry or other external standards, possess incremental milestones, and does it include qualitative criteria? Finally, how does the metric fit the organization objectives: Are the metrics challenging but at the same time attainable? Are assumptions and definitions specified for excellent performance? Are the employees who must enact the activity being measured fully involved in the development of this metric? Has the metric been jointly fixed by you and your service customer?

Table 7.1 Metric quality assessment table

Table 7.1 can be used as a starting point for assessing the development and tailoring of good metrics that are specific to an organization.

The end results are that the organization must deliver services to customers on time and deliver increased revenue or a protected market position when the data and measures are associated. Targets achieved must be quality services with increased buyer satisfaction, minimized or reduced rework or contract responsibility, possess a high return on investment (ROI), and ensure that the organizational capacity in resources and equipment is present to deliver the services.

Analytic Metrics

Beyond assessing the value that Big Data brings to organizational performance and decision making, a corresponding question is the need for metrics to measure the usefulness of analytics and analytic tools. There are few metrics for assessing tools or assessing analytical techniques and mapping outputs (of tools) to outcomes (of analytical approaches). Specialized metrics for specific domains will be required to assess analytical approaches and take account of how domain knowledge is used to derive actionable information and intelligence. Additionally, new verification and validation techniques will be required to support the assessment of tool outputs and analytical method outcomes where the results may be Big Data in their own right.

Going Forward: What Should a Manager Do?

The basis for applying Big Data analysis will require that decision makers and executives develop comprehensive measures of performance associated with organization transaction activity. With today’s technology, this can be extended to a sweeping instruction—measure everything, or at least establish a defined transaction baseline for beginning states of the outcomes and activities you will assess. But do it before you begin collecting and analyzing the data! However, there is no universal standard approach to establishing baselines. The collection and observing tools and the range and complexity of information made available can vary greatly among service providers, services, and customers. This enormous variability ultimately thwarts a generic or simple set of metrics from being applied for spotting the trends and findings that will be important in the analyses. The steps a manager and executive are to follow are straightforward.

Set Baselines

In the unassuming language of services, a performance baseline is composed of metrics that can define acceptable or standard operational conditions of a service. Obviously, the baseline performance will be used as a comparison to identify changes that indicate anything from a problem to an opportunity. Baselines may in themselves provide early warnings regarding service demands, capacity limitations, and upgrades in input requirements. Aligning baselines with targets or other objects can aid in staying within parameters or discerning difficult service spaces that may not meet standards or compliance goals.

Improve and Expand Classification Analysis

Decision makers and executives are familiar with the concept of grouping. Similarities in clients, services, requirements, demands, locations, and huge numbers of other characteristics are used to envisage participation that the client for a service is a member of a group or class, and therefore will find the offered service attractive. The class or classification membership is a predictor and therefore hugely important.

Predicting and explaining responses is the desired outcome. Our methods of developing classification are robust. A simple example of the use of a familiar classification tree is a good example of the benefits of classification. If we want to devise a system for approaching customers we may use a variety of tools—income, location, age, previous purchases, and interests—to classify them according to purchase potential. Thus, someone who files income taxes (at a level) does not live in a desert, is over the age of 17, has previously purchased products related to sailing, and subscribes to a sailing magazine may be a candidate for a sailboat purchase. Categorization processes and procedures can be more positive (e.g., categorizing plants and coins), but the principles for obtaining classifications and benefits are the same.

A decision maker or analyst does not need to know in advance what categories may be important. One can permit the data or values of the customers or attributes that were measured to describe the grouping or the structure of the metrics representing the customers included in our data. This permits the metric to aid in determining what groups of observations are all near each other and from that likeness find either other similarities or data to aid in reasoning why they may be together. Thus, it is not sufficient to just realize that there are natural groupings; one must still determine why the customers are grouped by an observation (or multiple values) of the metrics that we have. Simply said, the data we collect, and metrics describing the service or customer, will do some of our analytical work for us.

However, the metrics and data do not do it all. There is a critical role for the business knowledgeable expert, manager, and executive. They must apply their organizational, service, and business knowledge. Managers must contribute knowledge that is complex and constructed from a deep understanding of an overall market, market segmentation, and opportunities or limitations that can vary by geography, culture, technology, economic and social trends that could impact service performance, and customer requirement. They must also understand the competition’s strengths and weaknesses, and plans. Finally, managers must have an essential understanding of their envisioned customers in detail. Ask what the customer does, wants, values, and how current or similar services are now being employed to deliver value.

This complex understanding can be considered to be domain knowledge. And a manager or executive does not simply collect or acquire this understanding of the meaning of the Big Data but interprets, tests, predicts, and assembles all or portions of the Big Data to create a coherent depiction of the situation from the Big Data that are available for the analysis.

As an example, consider the expanding field of biology and the importance of Big Data. It is another useful Big Data domain with significant current research and great promise, but a deep technical, chemical, and biological understanding of this domain is essential for obtaining value from the analysis of Big Data. This domain covers the analysis of metabolites and biological molecules that are not as predicted or expected in terms of a location or quantity and the assessment of metabolites, macromolecules, proteins, DNA, and large molecule drugs in biological systems. Big Data analytical methods are available for measurable evaluation in biopharmaceutics and clinical pharmacology research. The data are used to assess the sample preparation of drugs in a biological matrix and to analyze the selectivity, specificity, limit of detection, lower limit of quantitation, linearity, range, accuracy, precision, recovery, stability, ruggedness, and robustness of liquid chromatographic methods. Researchers can then use Big Data tools to understand and report in studies on the pharmacokinetic (PK), toxicokinetic, bioavailability, and bioequivalence of a drug (Tiwari and Tiwari 2010).

It is readily apparent that this domain requires highly specialized knowledge to obtain value from these Big Data. Without such knowledge how can one possibly know or appreciate the relevance of these data? The executive, managers, and analysts of the future will have far more metrics available, but will need the classifications, knowledge, and insight to interpret and explain the associations. Thus, managers must invest in classification tools and expect more and better analysis as the groupings and findings fall out of the metrics and data. However, there is still an essential role for the manager!

Expect to Merge and Mix Data Types

The era of Big Data will see more and more to mixed data types. The events and activities of the world are experienced in conjunction with many other activities. Thus, the combined data may contain many types of information and classifications. Businesses and organizations have experienced this problem for many years. But the size and complexity of the data now make this problem more acute.

Mixed data can be best understood when one thinks of various categories. Some may be categories such as different types of service. Others may be continuous such as measurements showing sizes or quantities that can increase or decrease. Decision makers may expect that data will simultaneously combine data that have both categories and sizes—many of each at the same time. Differentiating and analyzing why metrics are obtained under each situation, and how changes will happen in the future, is the role of the manager and decision maker.

Examples will show that we can have an item or event categorized (as a yes or no, present or not present property); this could be combined with data that are frequency counts of how many times something happened. Medical situations can have these characteristics where patients are categorized by age, height, weight, test results, and previous occurrences of the event. Under these circumstances, the data can simply be counted, continuous, ranked, or as present or not present. Decision makers will need to be prepared for the volume of data, number of tools, results, and mixtures of data. Undeniably, all will increase as Big Data increases.

Executives: Listen and Watch Carefully for the Breakthroughs and Success Stories

It is difficult to envision the types of changes that will be encountered because of the growth of Big Data stimulated by the attraction, swiftness, and unpredicted impacts of social communications. The Big Data sets will also be added to and combined with the numerous sensor-driven data collections and ubiquity of enabled devices.

Many stories and articles of Big Data findings, applications, and successes will stimulate ideas and analytical approaches. Know and watch the competition, customers, services, and related industries for changes and breakthroughs via cross-industry pollination. A dramatic example of the use of mixed data detected structural changes on two publicly available social network data sets: Tactical Officer Education Program e-mail Network and open source Al-Qaeda communications network. Significant changes in both networks were detected with Al-Qaeda network structurally changing prior to the attacks of September 11 (McCulloh et al. 2008). Changes of this nature, that really make a service difference, will likely be exponential rather than incremental and similar to the black-swan event. One cannot be fully prepared, but do not be surprised.

What Is the Big Data Future?

All predictions point to a very bright future for Big Data! Obviously, executives and managers will encounter a great proliferation of new analysis tools, and trials of cleverly crafted organizational applications of this technology. Further, there are many educational programs churning out well trained and prepared students of the Big Data technology. They will have an enormous business impact. Organizations will hire and then quickly learn to strategically respond to the market with even more inventive uses of tools and combinations of data. Thus, there is no time to waste in applying the poser that can be drawn from Big Data to service delivery. What seems to be approaching, or may already be upon us, is a storm of changes (or an “arms race” like proliferation of technology) driven by Big Data.

This future is supported by a look at past predictions that help us understand the rapidity of the Big Data driven changes upon us. For a number of years, research firms and consultants predicted the rise and value that could be derived from Big Data and analytics. For example, McKinsey studied Big Data in five domains—health care in the United States, the public sector in Europe, retail in the United States, and manufacturing and personal-location data globally. Their findings lead to the prediction that Big Data could generate value in each area (Manyika et al. 2011).

With the growing attention on Big Data, the Pew Research Center began assessing it as a topic of interest in 2012 when the National Science Foundation and National Institutes of Health began seeking new methods, infrastructure, and approaches to find scientific and technological means for working with large data sets to accelerate progress in science and engineering, associated education, and training. The report described how Big Data is already having impacts with descriptions of spelling changes in Google search queries attributed to previous queries on the same subject that employed different spelling—a finding that enables an ability to identify search trends that may predict economic and public health trends (Anderson and Rainie 2012).

Other 2012 example uses include calls about “unusual activity” based on assessments of transaction anomalies in buyer behavior that may be evidence in indicators of fraud, and subscriber movie recommendations based upon profiles and previous actions of users. But there were still questions in the minds of managers and executives regarding effectiveness. Pew also reported that a 2012 survey of chief marketing officers report that more than half of the respondents said they then lacked the tools to mine true customer insights, and 58 percent lacked the skills and technology to perform analytics on marketing data (Anderson and Rainie 2012).

Today, as shown in this work, we can see that many predictions of the growth and uses of Big Data were correct. But what of the future for technology associated with Big Data, the public’s opinion and acceptance of the collection and uses of Big Data, and specific evolvement of business applications and uses?

In the area of analytics and technology, it is predicted that Hadoop will address many different scenarios and applications that require a mix of batch, real-time, streaming, and interactive scenarios (Taft 2014). However, limitations of the Hadoop MapReduce paradigm are already leading to new programming and analytics paradigms, such as Spark, Storm, and so on. From a hardware perspective, systems will use photonics for interconnections, memristers for huge amounts of storage that can be placed near the processing engines, and optimized computer and analytic processors for specialized tasks (Shaw 2014).

In the public space dealing with acceptance and understanding at large, the Big Data explosion has generated societal issues related to privacy, security, intellectual property, and liability. This may well lead to public policy and regulation beyond those current in place. Further, the value of the data may begin to be formalized as organizations learn and conduct transactions that determine in the marketplace what all the information is worth. There may be implied or explicit guidelines for assessing data value because data becomes recognized as an asset and can be equated to cash as standard methods for valuing data are developed. (And with the costs to society, perhaps we can expect new forms of taxation of this real, but difficult to measure, asset.)

From a business perspective, companies will continue to invent new models, search for associations in the data, and rapidly integrate information from multiple data sources, purchased from secondary sources and third parties. The sources and elements that will be added will include all forms of social media, volumes of sales and commercial data, location data, and accessible financial data. It is further predicted that users (subject matter experts) will create their own 360° views of data; the users will choose the data sources, analysis tool and technique, and the visualization—and will then be able to repeat this analytic as an experiment (as the gaming industry does today); organizational data will be structured and then connected to human interaction data; will improve our understanding of workers—state, mood, goals, preferences—and will recommend how to assist the human doing the work; and the tools will look for patterns in the data and will offer predictions of events for the users to asses and address (Shaw 2014).

Our Conclusion

Big Data has been a physical science problem for many years but has now become the “new” business and social science frontier. The amount of information and knowledge that can be extracted from the digital universe is continuing to expand as users come up with new ways to message and process data. Moreover, it has become clear that “more data is not just more data,” but that “more data is different,” for example, more diversity may produce better results.

Big Data is just the beginning of the problem. Technology evolution and placement guarantee that in a few years more data will be available in a year than has been collected since the dawn of man. If Facebook and Twitter are producing, collectively, around 50 gigabytes of data per day, and tripling every year, within a few years (perhaps three to five) we are indeed facing the challenge of Big Data becoming really Big Data.

There are numerous issues and challenges associated with Big Data usage. Technical scaling challenges will need to be solved as the volume of data approaches the 3Es—exabytes, exabits, exaflops—thresholds. However, the complexity of data—its complex semantics and structures, the context required to understand the data, and the need to track the provenance of derived data back to the original data—pose much larger problems.

The importance of Big Data to business and organizations—both profit and non profit—is indicated by the increasing number of Big Data startups each year. And more companies are engaging in Big Data initiatives either in-house or through consultants. Increased demand is putting pressure on the pool of data and analytic scientists. Training for data and analytic scientists will need to increase in both academic and corporate and organizational venues, with in-house training possibly becoming the training of first resort, at least for large organizations.

We conclude that the impact will be great, and the Big Data business itself will continue expanding. Research and Markets have predicted the evolution of Big Data technologies, markets, and outlays (expenditures) over the next five years—from 2015 to 2020. Using a triple-scenario analysis, including “steady state,” the emergence of new Big Data analytical tools, and the rise of new analytical tools that can replace Big Data analysis; their prediction is that the market will almost double in the coming five years, from nearly $39 billion in 2015 to more than $76 billion in 2020 (PRNewswire: Dublin 2014).

Our conclusion: Big Data is here, evolving, and swiftly becoming a major force that must be understood and effectively employed.