Chapter 3
WHAT ARE THE TRENDS IN BIG DATA?
LEARNING OBJECTIVES
After completing this chapter, you should be able to do the following:
• Distinguish between different Big Data trends.
• Identify how organizations are using Big Data and what their views are of Big Data going forward.
INTRODUCTION
What are the trends in Big Data? How are corporations reacting and planning to use Big Data in the coming years? An organization could gain a significant competitive advantage if it knows how to access and use Big Data. The trends in this chapter should be contrasted with your organization’s Big Data approach and used as benchmarks in the company’s strategic planning sessions.
TOP BIG DATA AND ANALYTICS TRENDS FOR 2017
Let’s consider some of the trends impacting 2017 and the future. We will then consider trends that were highlighted in the previous edition of the course.
The activity surrounding Big Data continues to increase. In the following sections, we will attempt to identify some of the current trends for 2017, as well as recognize trends that were prevalent over the last couple of years. Professionals will identify the following as possibilities to understand and exploit. Businesses that choose to ignore the potential of Big Data risk falling behind their competitors and possibly not having the ability to catch up. Tableau produced a list of significant trends for 2017:1
1. Modern business intelligence (BI) becomes the new normal as it moves away from IT-centric reporting and is a tool for everyone in the organization, not just IT.
2. Collaborative analytics, rather than traditional one-directional information creation and dissemination, will generate new insights. Organizations, departments, and individuals will work together and create joint meaning from formerly disparate data sources.
3. Data of all shapes, sizes, and forms will be explored. The data environment will not be limited to spreadsheets and individual datasets, but will include a variety of data types from structured, semi-structured, and unstructured data.
4. Non-analysts will become involved in data preparation as the move to self-service data occurs. Deriving meaning from data will become a component of many jobs (not just restricted to accounting, engineering, or IT.) Non-analysts will be able to access, manipulate, analyze, and report on Big Data.
5. Analytics will be available to people at all levels of the organization. It will most likely become hidden in plain sight. Just as e-mail is a regular tool of business, Big Data and analytics will not be thought of as separate functions.
6. IT moves from producer to enabler. IT will develop systems that allow non-analysts to access information without requiring a completely new skill.
7. Data analysis becomes more intuitive with drag-and-drop applications rather than scripting or pivot tables. Analytics will be easier to use and therefore more readily adopted by non-analysts.
8. Transition to the cloud accelerates. The migration to the cloud will continue alleviating much of the maintenance formerly required for infrastructure and software.
9. Advanced analytics will move to the user ranks. Non-analysts will not only be able to access information but will be able to perform more complex analytics.
Trends From 2016
1. Smart machines—Computers, sensors, and the like are being created to set up data from Big Data sources, interpret that data, and take action on that data.
a. Automating and removing human control—The processes are such that software is automating the identification, harvesting, analysis, and action phases so that human monitoring and intervention will not be necessary.
b. Augmenting decision support or wearable technology—Big Data tools will supplement existing decision support and wearable technology. Consider how pharmacists could improve quality control (and patient safety) by ensuring that the meds they supply have no harmful interactions. It would also be possible to integrate physician records with pharmacy databases and compare to national benchmarks to ensure better efficacy and patient welfare. Another simple example is the wearable Fitbit device, which has an electronic signal that measures whether daily milestones have been achieved. When the milestone is reached, the Fitbit buzzes, and users know that they have reached a critical metric to support overall health or well-being.
2. Customer digital assistants—Big Data allows quicker recognition of related data that could improve the respective transaction processing.
a. Systems that allow facial recognition and voice identification can be used to identify customers, employees, vendors, criminals, and so on. Emotion detection and natural language processing can reveal situations that can be detrimental or positive. Any opportunity to avert an unwanted situation would be welcomed by all parties. Also, positive instances can be recognized that showcase excellent customer or stakeholder interaction.
3. Machines authoring content—Big Data applications have resulted in computers venturing into journalistic areas. Computers are capable of writing:
a. Reports, white papers, press releases, articles and the like.
b. Robot writers are currently preparing thousands of fantasy football newsletter stories.
c. Expect advances in business content assistants such as Siri and Cortana. As computers become more advanced, personal software assistants will be able to integrate with Big Data analysis to enhance data query and processing instructions.
4. Robo Management
a. For repetitive positions, it is foreseeable that management will be robots. If the repetitive position can be benchmarked, a computer will be able to analyze, point out deviations from expectations and provide training and assistance to improve performance. This ability will be enhanced by computers that have self-learning capabilities.
5. Internet of Anything—The concept of Big Data included the "Internet of Things" (IoT), which means sensors and streaming data. This idea will morph into the "Internet of Anything." Disparate information generators will be accessed, aggregated and interpreted to produce new understanding.
a. The expectation from management will be to draw value from everything.
b. Examples of data that could be integrated into key metrics and financial information are oil rigs, satellites, weather buoys, GPS, and so on.
c. Unfortunately, the tools are not as robust as required, so additional development is expected to integrate all different levels of Big Data sources.
6. Big Data simplification—Even though Big Data discussion has been around for over 5 years, it has required a certain degree of expertise.
a. All users of Big Data are looking forward to simplification of the tools for access, integration, and analysis to create greater value.
b. Security—As seen by recent breaches involving Guccifer, Edward Snowden, and Julian Assange, companies have an enormous risk in the data maintained in their systems. The advancements in Big Data will only increase the potential exposure. Expect organizations in the future to improve their processes and systems to be in compliance with laws and regulations. Companies will also define how they will use Big Data, including information that has been captured or created based on customer or stakeholder information.
7. In 2016, Big Data will be used to create more customized applications for end users. Application developers will use data and analytics to create personalized, engaging experiences. The applications will try to unite related data across industries such as sports, energy, social well-being, and music. As an example, users will be able to select music based on personal preferences, such as instruments and tempo.
a. As part of the user-generated data discovery—friendlier end-user tools will be made available to allow less sophisticated users to participate in Big Data analysis. The number of software tools such as Tableau, Qlikview, and Excel BI will increase. This effort will become known as self-service data analytics
8. Hadoop will allow mission critical applications that weren’t previously available. Hadoop is a tool that enables access to disparate databases. Due to increased usage and demands, expect to see Hadoop usage grow and become applied to central company applications.
9. The growth of Big Data will also result in the personalization of applications for the end user. Early in the development of the Internet, end users could create their home portal with critical data that they were interested in, such as weather, news, and stock prices. This new trend will allow end users to access, mine and predict Big Data outcomes just as if they were asking for the weather or the news.
10. Apache Spark is moving from a component of the Hadoop software to the Big Data platform of choice.
11. Although the original vision of Big Data was to house it internally, major Big Data companies such as Google, Amazon, and others are integrating Big Data, the cloud, and IoT. Just as they were key players in migrating to the cloud, they are making the same inroads with organizations moving their Big Data efforts from internal to the cloud.
12. Executive teams will embrace algorithms as tools to increase or generate value. The strategic objectives will include efforts to identify the value-adding algorithms.
13. There is a need to act quickly to access, analyze, decide and act before Big Data loses value and the company loses competitive advantage. At a bare minimum, companies in 2016 must address Big Data during their strategic planning sessions.
14. Big Data will allow improved access to image recognition and natural language understanding. Also, it should provide the potential to increase deeper learning and understanding than what is currently achieved.
a. The expansion of Big Data capabilities will provide more context around data such as demographics and location.
15. Big Data will migrate away from simple data dumps to improved access and mining of existing databases.
16. As Big Data evolves, professionals will be required to understand and manipulate data to their advantage. The role of data scientist will be incorporated into many existing job descriptions.
17. Machine learning will increase and replace traditional manual trial and error that humans currently attempt.
a. Expect Big Data devices to collect, analyze, and store data without human intervention. For those attendees who love movies, consider the examples of Danny DeVito’s computer to search out breakup candidates in Other People’s Money, or the psychopathic computer HAL in the movie 2001: A Space Odyssey.
18. There is a good chance that the term "Big Data" may go by the wayside and descriptions will be focused on the specific segments of collecting, storing, mining, algorithms, and so on.
19. According to the Bureau of Labor Statistics, there is a shortage of skilled data scientists for the next several years.
20. MPP (multi-parallel processing), which was touted as being an in-house system, will most likely move to the cloud.
The trends in the following sections were highlighted in the first edition of this course, released in the summer of 2015. The trends were identified via surveys and studies performed by the following leading consulting firms:
• Gartner & Co.
• Accenture
• NewVantage Partners
BIG DATA SURVEY
Gartner & Co. is a leading consulting company specializing in information technology.
Gartner believes the rapidly evolving modern BI and analytics market is being influenced by the following seven dynamics:
1. Modern BI tools that support greater accessibility, agility, and analytical insight at the enterprise level will dominate new purchases.
2. The emergence of smart data discovery capabilities, machine learning, and automation of the entire analytics workflow will drive a new flurry of buying by established vendors because of their potential value to reduce time to insights from advanced analytics and deliver those insights to a broader set of people across the enterprise.
3. There will be a larger investment in data preparation of complex datasets as business users want to analyze increasingly large, diverse, and complex combinations of data sources and data models, faster than ever before.
4. The ability to embed and extend analytics content will be a key enabler of more pervasive adoption of and ability to gain value from analytics.
5. Organizations will increasingly leverage streaming data generated by devices, sensors, and people to make faster decisions.
6. Cloud deployments of BI and analytics platforms have the potential to reduce the cost of ownership and speed time to deployment. Gartner expects that, by 2020, the majority of new licensing buying is likely to be for cloud deployments.
7. The availability of an active marketplace where buyers and sellers converge to exchange analytic applications, aggregated data sources, custom visualizations, and algorithms is likely to generate increased interest in the BI and analytics space and fuel its future growth.
Organizations will benefit from the many new and innovative vendors continuing to emerge.2
In other studies, Gartner also pointed out the following:
Increased Use of Drones
Production of drones for personal and commercial use is growing rapidly, with global market revenue expected to increase 34 percent to reach more than $6 billion in 2017 and grow to more than $11.2 billion by 2020, according to a new forecast from Gartner, Inc. Almost three million drones will be produced in 2017, 39 percent more than in 2016.
The market for commercial drones is much smaller than the consumer market, with a significantly higher average selling price in comparison with personal drones. With more countries solidifying their drone regulations, the market is beginning to stabilize, and companies are now buying drones to test and deploy in nearly every industry. Commercial drones normally have a higher payload, longer flight times, and redundant sensors and flight controllers to make them safer. They are more specialized to a function, such as mapping, delivery, or industrial inspection, so prices vary according to these requirements.3
Connected Internet Things
Gartner, Inc., forecasts that 8.4 billion connected things will be in use worldwide in 2017, up 31 percent from 2016, and will reach 20.4 billion by 2020. Total spending on endpoints and services will reach almost $2 trillion in 2017.
Regionally, Greater China, North America, and Western Europe are driving the use of connected things and the three regions together will represent 67 percent of the overall IoT installed base in 2017.
The consumer segment is the largest user of connected things, with 5.2 billion units in 2017, which represents 63 percent of the overall number of applications in use. Businesses are on pace to employ 3.1 billion connected things in 2017.4
KNOWLEDGE CHECK
1. Gartner forecasts that how many connected things will be in use by 2017?
a. 4.4 billion.
b. 6.8 billion.
c. 8.4 billion.
d. 9.6 billion
ACCENTURE TRENDS AND SURVEYS
In March of 2017, Accenture conducted a study that concluded that nearly half (48 percent) of CFOs believe digital technologies will fundamentally change everything finance does.5
Some of their other findings include the following:
• Eight in 10 CFOs are seeing measurable return on investment (ROI) from digital finance investments.
• One in four companies realize greater than expected returns from digital, and 82 percent are seeing measurable business ROI from digital finance investments. In addition, finance organizations reported the following:
— 86 percent can better manage risk.
— 67 percent have improved their forecast accuracy.
— 66 percent report better decision making.
— 61 percent say finance teams are dedicating more time to high-value work.6
• Today’s CFOs are embracing digital more than ever. It is much more than social, mobile and analytics. CFOs are increasingly relying on digital for
— security threat intelligence,
— blockchain, and
— artificial intelligence.
• Digital is currently delivering improvement in professional staff productivity and could soon make monthly, and quarterly, management reports a relic of a bygone time.
• About one-third of all digital technologies have delivered transformational benefits to firms with cycle time reduction (37 percent) and finance staff productivity improvement (36 percent) being the top benefits.
• The study also highlighted a few barriers:
— The level of investment required for digital technologies (18 percent)
— Hiring talent that has the required skills to implement and operate the new technologies (16 percent)
In an earlier Accenture study entitled, "Big Success with Big Data," the consulting firm listed the following five major findings:
1. Big Data is taking off. Users who have completed a Big Data project are satisfied with the results.
2. Bigger companies are getting more from Big Data. Companies are achieving a more significant payback.
3. Big Data demands broad learning. Organizations will have to invest in training tools.
4. Expert outside help will be needed. It is difficult to find skilled in-house talent.
5. Big Data is disruptive but potentially transformational.
KNOWLEDGE CHECK
2. Users who have completed a Big Data project are ___________ with the results.
a. Ambivalent.
b. Dissatisfied.
c. Satisfied.
d. Ecstatic.
3. According to Accenture, what is the most widely accepted characteristic of Big Data?
a. Advanced analytics.
b. File sizes > 20 TB.
c. Data from social networks.
d. Data from machines.
4. What percentage of respondents strongly agree that companies will lose competitive position if they do not embrace Big Data?
a. 51 percent.
b. 39 percent.
c. 37 percent.
d. 62 percent.
NEW VANTAGE PARTNERS 5TH ANNUAL BIG DATA EXECUTIVE SURVEY8
According to NewVantage Partners 5th annual Big Data Executive Survey, 48.4 percent of corporate executives who were surveyed indicated that their firm has achieved "measurable results" from their Big Data investments. Further, a remarkable 80.7 percent of executives now characterize their Big Data efforts as having been successful.
The survey highlights the challenges that major corporations still face as they seek to become more data-driven organizations:
• 85.5 percent of executives report that their organization has taken steps to create a data-driven culture, but only 37.7 percent report that these efforts have been successful to date.
• 52.5 percent of executives report that organizational impediments, including lack of organizational alignment, business or technology resistance, and lack of middle management adoption are factors limiting the success of Big Data efforts.
• 60.7 percent of executives report that their firm has developed an enterprise Big Data strategy, but 18 percent report that their firm lacks "a coherent data strategy."
• 95 percent of executives report that their organization has undertaken a Big Data initiative within the past five years.
• The survey shows a range of Big Data initiatives:
— Efforts to decrease expenses through operational cost efficiencies have proven to be successful (49.2 percent) for many firms.
— Only 27.9 percent of respondents believe they have achieved a data-driven culture.
— New avenues for innovation and disruption have had the highest success rate—64.5 percent started, 44.3 percent reporting results, 68.7 percent success rate.
KNOWLEDGE CHECK
5. According to the NewVantage Partners survey, approximately what percentage of respondents reported success with Big Data efforts to achieve cost efficiencies by reducing expenses?
a. 10 percent.
b. 30 percent.
c. 50 percent.
d. 70 percent.
CSC STUDY—BIG DATA AND ANALYTICS
CSC (Computer Sciences Corporation) prepared a study entitled "Big Data & Analytics—From Sensory
Overload to Predictable Outcomes."9 The survey highlights CFO and CIO expectations from Big Data
and how to gain a competitive advantage by mastering Big Data.
The major points of the survey were the following:
1. Data are a competitive tool for any organization. However, it must be the correct data, properly managed and arranged, and must come from a wide range of sources.
2. Data must be transformed into actionable business insights.
3. Eager enthusiasts use Big Data for historical perspectives.
4. Half of those surveyed believed that Big Data could have as much of an impact on their organization as the World Wide Web did.
5. The majority of CFOs say that the quality of data and the speed at which it is delivered is adequate.
6. Most CFOs see Big Data as a cost.
7. To successfully take advantage of Big Data, business and information technology development must work together.
8. More than half of the respondents employed data scientists, mathematicians, or pattern trackers.
9. There must be one individual who is charged with the responsibility of bringing together large amounts of data successfully from a variety of sources as part of the strategic decision making for the organization.
Practice Questions
1. In which industries does Tableau believe that Big Data will create more customized applications?
2. What does Accenture conclude about internal talent for Big Data projects?