Introduction

“It is possible to fly without motors, but not without knowledge or skill.”

-- Wilbur Wright

65 million years ago an asteroid hit planet Earth and the dinosaurs were suddenly doomed! A massive extinction event followed, yet as the saying goes, there is a silver lining to every cloud. This horrific event paved the way for a new era, where creatures called "mammals" began to rule. This led to the evolution of man and woman kind, the rise of prophets, and eventually computers.

65 months ago "Big Data" hit planet Earth and hardware and software dinosaurs were suddenly doomed! A massive extinction event will follow, yet there is gold in cloud computing. This paradigm-changing event has paved the way for a new era, where companies that are "customer centric" have begun to rule. This will lead to the evolution of products and services for men and women customers, a rise in profits, and computer cost savings!

Dinosaurs couldn't predict the asteroid -- and many hardware and software vendors didn’t see the Big Data impact coming!

This book is a wake-up call! The alarm has just gone off; the objective of this book is to help you gain a deeper understanding of Big Data and Hadoop. This will help to gauge its impact, and the changes you need to not only survive, but to thrive in this new era!

History of Data Warehousing

“Live out of your imagination, not your history.”

-- Stephen Covey

When the IBM mainframe became an integral part of the IT foundation for many companies, the industry went through a transformation equivalent to humans growing from infancy to adulthood. Companies built applications that would support running their businesses. The equipment expenses were high, as were the maintenance costs, yet as has often been said, “No pain, no gain”.

Then a little company called Intel emerged. Intel produced an innovative chip called the 8086 that was used in early personal computers (PCs). The PC became one of the greatest inventions of all time. No one knew that incredible innovations, such as cures for diseases, would one day arise from this invention.

While companies continued using the mainframes for most computing needs, the PC started to enter into companies on employees’ desktops. However, the desktop PC was used mainly for administrative and managerial tasks.

A significant leap forward in computing occurred in 1979 as a small company called Teradata emerged. Their concept was to take the Intel 8086 chips and string them together over a network. Each chip would have its own operating system, memory and disk. The system would then take data and spread it equally among the processers. Parallel processing was born and this baby came out big, hungry, and aggressive. Teradata figured that if they could string enough of these chips together the system could be more powerful than a mainframe at significantly less cost. IBM and industry experts were not impressed. The industry opinion may have been summed up with the following analogy: The concept made about as much sense as a farmer plowing a field with a thousand chickens.

However, nobody is laughing now. Teradata got the proverbial chickens coordinated and was able to dig up fields of data faster than was thought possible. Teradata didn’t create their parallel processing engine to run the business, rather instead to analyze massive amounts of data in order to make decisions. This was the foundation for the invention of the data warehouse.

The Growth of Computer Data and Use of Databases

“The more he gives to others, the more he possesses of his own.”

– Lao-Tzu

The usage of computing continued to evolve, with a focus on managing daily operations by using transactional systems. Initially the transactions were moved to databases on a monthly schedule. Over time the frequency of database loads increased to weekly, and eventually many companies moved to a daily process. Today many large companies move datasets every 15 minutes.

One large national retailer keeps 66 weeks of data on each product they offer. This retailer creates an effective data warehouse strategy, with a roadmap of improvements over time. This approach sets what could be considered a best practice for the data warehouse industry. The evolution of data warehousing has been helped by the collaboration of business and technology teams. Innovations have supported business intelligence, allowing users to access data efficiently to identify focus areas and perform data mining.

In February 2010, The Economist published a report titled “Data, data everywhere”. Kenneth Cukier wrote “the world contains an unimaginably vast amount of digital information which is getting ever vaster more rapidly . . . Scientists and computer engineers have coined a new term for the phenomenon: “big data.”

According to various experts, the amount of data being stored now is growing exponentially. A Gartner study projected 800% growth within 5 years. Is your company ready to handle this level of data growth? Is your company prepared to manage an exponential growth rate for un-structured data?

Enterprise Data vs. Big Data

“When love and skill work together, expect a masterpiece.”

– John Ruskin

Until recently the data warehouse was where all enterprise data could be analyzed, yet no system was powerful or scalable enough to hold the massive amounts of data companies consume and analyze. All of the data inside an enterprise should be considered as part of a distributed and logical data warehouse.

Big data isn’t just more of the same data, it encompasses different types of data and evolving classes of data. We now have more than tables with columns and rows. We have pictures, videos, contracts, weather reports, GPS location and tracking devices, social media and more. Traditional systems weren’t designed for this originally, however due to the parallel processing efforts of yesterday, the path has been paved to meet the evolving needs of today.

Cloud computing, commodity hardware and the emerging concept of “virtualization” enables company’s cost-effective ways to store and manage enormous amounts of data for analysis. The chips are not 8086 chips anymore, the memory is stable, and the disks are incredibly powerful. There is little that we as humans can no longer track and analyze. This is the era of Big Data!

Why is Big Data Important?

“A bird does not sing because it has the answers, it sings because it has a song.”

-Anonymous

Big data does not sing because it has a song, it sings because it has the answers! Big data allows for decisions to be made based on facts. Cross-analysis can be performed and the truth, the whole truth and nothing but the truth can be revealed and presented. Decisions can then be made with confidence - this can make anyone sing.

Why Enhance Your Company’s Data Warehousing Capabilities?

“That's a sparrow's belch in the midst of a typhoon.”

-Alan Simpson

Why Enhance Your Company’s Data Warehousing Capabilities? (“We already have standard reports we run monthly.”) Your company’s future depends on your ability to adapt to a changing data environment. The competitive marketplace demands that companies become more customer-centric. As data volumes increase, business intelligence will have to evolve to handle Big Data that can include massive amounts of un-structured data.

Benefits of Big Data for Your Company

“I don’t know who my grandfather was. I am more interested in who his grandson will become.”

– Abraham Lincoln, President of the United States

It has been said, “Those who don’t study the past are condemned to repeat it.” Big data is the enabler that every enterprise needs to build upon the good things from the past, and can also be used to avoid repeating the mistakes of the past. You have the opportunity to look into the data to see your customers, better understand their needs, and identify products and services to improve their life. Take the example of an existing or potential customer who has moved to a new town; that information can be used to deliver products or services that are customized for their needs. If the customer is traveling, target specific travel related services that will enhance their trip.

Imagine that you are traveling on a ship, and have fallen overboard. Consider Big Data as a life raft that is within your grasp – a unique opportunity to convert your current hardware, tools and approach into being a survivor! If I were to fall overboard on a ship, and then have someone approach me on a lift raft and save me, I would say “thank you!” If that same person then asked if I'd like to thrive after surviving, I'd say "tell me how!". This book provides background information on how to survive and thrive!

In every industry right now the trends are evolving. However, if you are only concerned about your current products and how you’re selling those products, you are missing out on tremendous opportunities. If you want to know your customers, get real-time feedback from customers, then Big Data can unleash the power that may lie un-tapped within your organization’s reach. Improvements can even be made with your traditional data, due to being able to leverage data sources from outside the company. This ability to tap into data to inform business decisions, and drive improvements to meet evolving customer needs, is what Big Data brings to your company.

Management Considerations

“I saw the angel in the marble and carved until I set him free.”

--Michelangelo

The mid-level and front-line managers’ jobs involve executing on the vision set by senior management. These managers must take the direction and strategy from senior leaders locally or in headquarters locations - and make it a reality - this is both a science and an art. Senior management may expect the artistic equivalent of a masterpiece from the Sistine Chapel. From an operational perspective, mid-level and Front-line managers must translate that artistic vision into a functional business that efficiently uses resources to meet customer needs in a profitable way.

As the enterprise commits to a big data strategy, these managers will need a solid understanding of what this means to their organizations. These managers will be the ones who setup the environment, and hire and guide the people. They need to enable processes, people and plan for the changes ahead. Managers need to create an environment that supports business intelligence, and link this to better services, products and processes. It is the managers who start with a proverbial marble block and carve until they make angels, or products that meet their customers’ needs.

Customer Considerations

"You can make more friends in two months by becoming interested in other people, than you will in two years by trying to get other people interested in you."

Dale Carnegie

Few people like to get spam emails, or marketing brochures, or sales texts, unless it is exactly what they need at the time which is not common. What was important to a customer yesterday may not be important today, and what is important today may be less important tomorrow. You can make more customers in two weeks by selling products that they are interested in, than you will in two years by trying to get customers interested in products that are interesting to you.

Big data allows companies to find out what a customer is interested in at a particular time. Have they just announced on Facebook that today their first child was born and that this is the greatest moment of their life? Or did they mention on Twitter about a University alumni event they plan to join? Did the customer provide reviews on products and services? Does the customer provide in-app feedback on new features? Do you collect travel information on customers that could be cross-referenced for various marketing offers? All these types of data can be leveraged for insights by using Big Data.

Today’s customers purchase a product suite from a company that understands what they want. It is personalized for them. If you can provide products relevant to them based on Big Data you customize their experience. Remember, every customer is unique!

Joachin and his wife were visiting a small town near where they live in Florida, planning to have dinner at a local restaurant that evening. In the afternoon they drove by a restaurant called Ibiza’s and his wife commented that Ibiza is a Spanish Island. Later on when they were deciding where to eat, his wife got a sudden craving for Spanish food. They went to Ibiza’s restaurant and the meal was great. Later that week Joachin opened an online offer and it was a deal for a complete package trip to Ibiza from a travel service, with an all-inclusive stay for 5 days.

He bought the travel package, surprised his wife and was inducted into the husband hall of fame! In the past companies had to throw marketing darts, yet now with the power of big data they can offer relevant marketing to beat the competition, make the sale and thrill the customer.

An Industry Example

“Darkness cannot drive out darkness; only light can do that. Hate cannot drive out hate; only love can do that.”

– Martin Luther King, Jr.

The auto industry has turned the light on Big Data and this revolution has transformed vehicles into computers on wheels. Vehicles are now integrating with GPS systems, and soon they will leverage Big Data to optimize customer needs. This includes monitoring tire pressure, driving route patterns, analysis of erratic maneuvers (such as starting and stopping too fast or swerving), and media analysis. The possibilities to customize the experience for an individual are limitless.

Innovation can take many forms. A future example may be smart traffic lights. Why sit at a red light when nobody else is on the road? Imagine real-time lights and the time saved and the gas savings. Another example is real-time driving adjustments: We’ll see cars that can avoid accidents by braking automatically or adjusting lane position due to the location of other cars.

Opportunities will continue to emerge improvements in the driving experience, ranging from identifying customer preferences on music, to vehicle maintenance, to recommendations on restaurants and shopping, to suggesting other activities or destinations.

The timing is good for the auto industry to give customers a break, as they shift gears towards Big Data to accelerate customer satisfaction into the fast lane!

What does it mean to be data driven?

“When I was 14 I thought my parents were the stupidest people in the world. When I was 21 I was amazed at how much they learned in seven years.”

Mark Twain

Sometimes liars figure, but figures never lie. The next time you are going on a long sales trip let the data drive! Enterprises have been built around applications that automate a process. An example is when management says: “Let's automate our billing system.” The company may have a process where the point of sale equipment shows something was ordered. They can develop processes so they can effectively collect their money. This is a billing system, but it is not data driven. It is built around a process to collect money. Data driven means that decisions are based upon analytics; this enables process changes.

Companies have much historical information about customers, and systems are built around the processes. However, combining the data on these legacy systems allow you to see data on customers, products they’re using, and enables asking customers what they want.

The sales system can then be aware of which customer is being called, and the person selling can let the data drive the sale. Under this scenario, offers that are relevant to the customer go to the customer.

The systems built around processes are important for understanding workflows, yet systems can be enhanced to capture new information, and enhance the customer experience. For example, if the data shows a pattern of fraud, that information can be utilized to modify processes to reduce fraud.

Apple was one of the first companies that designed product lines around greatly improving people’s lives. An example is around analysis of music listening habits. Imagine if they discovered what songs are playing when you turn up the volume or what songs make you change the channel. That’s an example of the possibilities with Big Data, where customer data analysis is key to making individual decisions.