The amount of data that's being created and stored on a global level is almost inconceivable, and it just keeps growing. Big data is a term that describes the large volume of data—both structured and unstructured. Let's now delve deeper into big data, beginning with the challenges of big data.
Big data is characterized by three challenges. They are as follows:
- The volume of the data
- The velocity of the data
- The variety of the data
The volume problem can be approached from three different directions: efficiency, scalability, and parallelism. Efficiency is about minimizing the time it takes for an algorithm to process a unit of information. A component of this is the underlying processing power of the hardware. The other component, and the one that we have more control over, is ensuring that our algorithms are not wasting precious processing cycles with unnecessary tasks.
Scalability is really about brute force and throwing as much hardware at a problem as you can. Taking into account Moore's law, which states that the trend of computer power doubling every two years, will continue until it reaches its limit; it is clear that scalability is not, by itself, going to be able to keep up with the ever-increasing amounts of data. Simply adding more memory and faster processors is not, in many cases, going to be a cost effective solution.
Parallelism is a growing area of machine learning, and it encompasses a number of different approaches, from harnessing the capabilities of multi-core processors, to large-scale distributed computing on many different platforms. Probably, the most common method is to simply run the same algorithm on many machines, each with a different set of parameters. Another method is to decompose a learning algorithm into an adaptive sequence of queries, and have these queries processed in parallel. A common implementation of this technique is known as MapReduce, or its open source version, Hadoop.
The velocity problem is often approached in terms of data producers and data consumers. The rate of data transfer between the two is called the velocity, and it can be measured in interactive response times. This is the time it takes from a query being made to its response being delivered. Response times are constrained by latencies, such as hard disk read and write times, and the time it takes to transmit data across a network.
Data is being produced at ever greater rates, and this is largely driven by the rapid expansion of mobile networks and devices. The increasing instrumentation of daily life is revolutionizing the way products and services are delivered. This increasing flow of data has led to the idea of streaming processing. When input data is at a velocity that makes it impossible to store in its entirety, a level of analysis is necessary as the data streams, in essence, deciding what data is useful and should be stored, and what data can be thrown away. An extreme example is the Large Hadron Collider at CERN, where the vast majority of data is discarded. A sophisticated algorithm must scan the data as it is being generated, looking at the information needle in the data haystack. Another instance that processing data streams may be important is when an application requires an immediate response. This is becoming increasingly used in applications such as online gaming and stock market trading.
It is not just the velocity of incoming data that we are interested in; in many applications, particularly on the web, the velocity of a systems output is also important. Consider applications such as recommender systems that need to process a large amount of data and present a response in the time it takes for a web page to load.
Collecting data from different sources invariably means dealing with misaligned data structures and incompatible formats. It also often means dealing with different semantics and having to understand a data system that may have been built on a fairly different set of logical premises. We have to remember that, very often, data is repurposed for an entirely different application from the one it was originally intended for. There is a huge variety of data formats and underlying platforms. Significant time can be spent converting data into one consistent format. Even when this is done, the data itself needs to be aligned such that each record consists of the same number of features and is measured in the same units.
Consider the relatively simple task of harvesting data from web pages. The data is already structured through the use of a mark language, typically HTML or XML, and this can help give us some initial structure. Yet, we just have to peruse the web to see that there is no standard way of presenting and tagging content in an information-relevant way. The aim of XML is to include content-relevant information in markup tags, for instance, by using tags for author or subject. However, the usage of such tags is far from universal and consistent. Furthermore, the web is a dynamic environment and many web sites go through frequent structural changes. These changes will often break web applications that expect a specific page structure.
The following diagram shows two dimensions of the big data challenge. I have included a few examples where these domains might approximately sit in this space. Astronomy, for example, has very few sources. It has a relatively small number of telescopes and observatories. Yet the volume of data that astronomers deal with is huge. On the other hand, perhaps, let's compare it to something like environmental sciences, where the data comes from a variety of sources, such as remote sensors, field surveys, validated secondary materials, and so on.
Integrating different data sets can take a significant amount of development time; up to 90 percent in some cases. Each project's data requirements will be different, and an important part of the design process is positioning our data sets with regard to these three elements.
A fundamental question for the data scientist is how the data is stored. We can talk about the hardware, and in this respect, we mean nonvolatile memory such as the hard drive of a computer or flash disk. Another way of interpreting the question (a more logical way) is how is the data organized? In a personal computer, the most visible way that data is stored is hierarchically, in nested folders and files. Data can also be stored in a table format or in a spreadsheet. When we are thinking about structure, we are interested in categories and category types, and how they are related. In a table, how many columns do we need, and in a relational data base, how are tables linked? A data model should not try to impose a structure on the data, but rather find a structure that most naturally emerges from the data.
Data models consist of three components:
- Structure: A table is organized into columns and rows; tree structures have nodes and edges, and dictionaries have the structure of key value pairs.
- Constraints: This defines the type of valid structures. For a table, this would include the fact that all rows have the same number of columns, and each column contains the same data type for every row. For example, a column,
items sold, would only contain integer values. For hierarchical structures, a constraint would be a folder that can only have one immediate parent. - Operations: This includes actions such as finding a particular value, given a key, or finding all rows where the items sold are greater than 100. This is sometimes considered separate from the data model because it is often a higher-level software layer. However, all three of these components are tightly coupled, so it makes sense to think of the operations as part of the data model.
To encapsulate raw data with a data model, we create databases. Databases solve some key problems:
- They allow us to share data: It gives multiple users access to the same data with varying read and write privileges.
- They enforce a data model: This includes not only the constraints imposed by the structure, say parent child relationships in a hierarchy, but also higher-level constraints such as only allowing one user named bob, or being a number between one and eight.
- They allow us to scale: Once the data is larger than the allocated size of our volatile memory, mechanisms are needed to both facilitate the transfer of data and also allow the efficient traversal of a large number of rows and columns.
- Databases allow flexibility: They essentially try to hide complexity and provide a standard way of interacting with data.
A key characteristic of data is its probability distribution. The most familiar distribution is the normal or Gaussian distribution. This distribution is found in many (all?) physical systems, and it underlies any random process. The normal function can be defined in terms of a probability density function:
Here, δ (sigma) is the standard deviation and µ (mu) is the mean. This equation simply describes the relative likelihood a random variable, x, will take on a given value. We can interpret the standard deviation as the width of a bell curve, and the mean as its center. Sometimes, the term variance is used, and this is simply the square of the standard deviation. The standard deviation essentially measures how spread out the values are. As a general rule of thumb, in a normal distribution, 68% of the values are within 1 standard deviation of the mean, 95% of values are within 2 standard deviations of the mean, and 99.7% are within 3 standard deviations of the mean.
We can get a feel for what these terms do by running the following code and calling the normal() function with different values for the mean and variance. In this example, we create the plot of a normal distribution, with a mean of 1 and a variance of 0.5:
import numpy as np import matplotlib.pyplot as plt import matplotlib.mlab as mlab def normal(mean = 0, var = 1): sigma = np.sqrt(var) x = np.linspace(-3,3,100) plt.plot(x,mlab.normpdf(x,mean,sigma)) plt.show() normal(1,0.5)
Related to the Gaussian distribution is the binomial distribution. We actually obtain a normal distribution by repeating a binomial process, such as tossing a coin. Over time, the probability approaches that half the tosses will result in heads.
In this formula, n is the number coin tosses, p is the probability that half the tosses are heads, and q is the probability (1-p) that half the tosses are tails. In a typical experiment, say to determine the probability of various outcomes of a series of coin tosses, n, we can perform this many times, and obviously the more times we perform the experiment, the better our understanding of the statistical behavior of the system:
from scipy.stats import binom def binomial(x=10,n=10, p=0.5): fig, ax = plt.subplots(1, 1) x=range(x) rv = binom(n, p) plt.vlines(x, 0, (rv.pmf(x)), colors='k', linestyles='-') plt.show() binomial()
You will observe the following output:
Another aspect of discrete distributions is understanding the likelihood of a given number of events occurring within a particular space and/or time. If we know that a given event occurs at an average rate, and each event occurs independently, we can describe it as a Poisson distribution. We can best understand this distribution using a probability mass function. This measures the probability of a given event that will occur at a given point in space/time.
The Poisson distribution has two parameters associated with it: lambda ,λ, a real number greater than 0, and k, an integer that is 0, 1, 2, and so on.
Here, we generate the plot of a Poisson distribution using the scipy.stats module:
from scipy.stats import poisson def pois(x=1000): xr=range(x) ps=poisson(xr) plt.plot(ps.pmf(x/2)) pois()
The output of the preceding commands is as shown in the following diagram:
We can describe continuous data distributions using probability density functions. This describes the likelihood that a continuous random variable will take on a specified value. For univariate distributions, that is, those where there is only one random variable, the probability of finding a point X on an interval (a,b) is given by the following:
This describes the fraction of a sampled population for which a value, x, lies between a and b. Density functions really only have meaning when they are integrated, and this will tell us how densely a population is distributed around certain values. Intuitively, we understand this as the area under the graph of its probability function between these two points. The Cumulative Density Function (CDF) is defined as the integral of its probability density functions, fx:
The CDF describes the proportion of a sampled population having values for a particular variable that is less than x. The following code shows a discrete (binomial) cumulative distribution function. The s1 and s2 shape parameters determine the step size:
import scipy.stats as stats def cdf(s1=50,s2=0.2): x = np.linspace(0,s2 * 100,s1 *2) cd = stats.binom.cdf plt.plot(x,cd(x, s1, s2)) plt.show()
We generally interact with databases via a query language. One of the most popular query languages is MySQL. Python has a database specification, PEP 0249, which creates a consistent way to work with numerous database types. This makes the code we write more portable across databases and allows a richer span of database connectivity. To illustrate how simple this is, we are going to use the mysql.connector class as an example. MySQL is one of the most popular database formats, with a straight forward, human-readable query language. To practice using this class, you will need to have a MySQL server installed on your machine. This is available from https://dev.mysql.com/downloads/mysql/.
This should also come with a test database called world, which includes statistical data on world cities.
Ensure that the MySQL server is running, and run the following code:
import mysql.connector from mysql.connector import errorcode cnx = mysql.connector.connect(user='root', password='password', database='world', buffered=True) cursor=cnx.cursor(buffered=True) query=("select * from city where population > 1000000 order by population") cursor.execute(query) worldList=[] for (city) in cursor: worldList.append([city[1],city[4]]) cursor.close() cnx.close()
Information on the web is structured into HTML or XML documents. Markup tags give us clear hooks for us to sample our data. Numeric data will often appear in a table, and this makes it relatively easy to use because it is already structured in a meaningful way. Let's look at a typical excerpt from an HTML document:
<table border="0" cellpadding="5" cellspacing="2" class="details" width="95%"> <tbody> <th>Species</th> <th>Data1</th> <th>data2</th> </tr> <td>whitefly</td> <td>24</td> <td>76</td> </tr> </tbody> </table>
This shows the first two rows of a table, with a heading and one row of data containing two values. Python has an excellent library, Beautiful Soup, for extracting data from HTML and XML documents. Here, we read some test data into an array, and put it into a format that would be suitable for input in a machine learning algorithm, say a linear classifier:
import urllib from bs4 import BeautifulSoup import numpy as np url = urllib.request.urlopen("http://interthing.org/dmls/species.html"); html = url.read() soup = BeautifulSoup(html, "lxml") table = soup.find("table") headings = [th.get_text() for th in table.find("tr").find_all("th")] datasets = [] for row in table.find_all("tr")[1:]: dataset = list(zip(headings, (td.get_text() for td in row.find_all("td")))) datasets.append(dataset) nd=np.array(datasets) features=nd[:,1:,1].astype('float') targets=(nd[:,0,1:]).astype('str') print(features) print(targets)
As we can see, this is relatively straight forward. What we need to be aware of is that we are relying on our source web page to remain unchanged, at least in terms of its overall structure. One of the major difficulties with harvesting data off the web in this way is that if the owners of the site decide to change the layout of their page, it will likely break our code.
Another data format you are likely to come across is the JSON format. Originally used for serializing Javascript objects, JSON is not, however, dependent on JavaScript. It is merely an encoding format. JSON is useful because it can represent hierarchical and multivariate data structures. It is basically a collection of key value pairs:
{"Languages":[{"Language":"Python","Version":"0"},{"Language":"PHP","Version":"5"}],
"OS":{"Microsoft":"Windows 10", "Linux":"Ubuntu 14"},
"Name":"John"the fictional" Doe",
"location":{"Street":"Some Street", "Suburb":"Some Suburb"},
"Languages":[{"Language":"Python","Version":"0"},{"Language":"PHP","Version":"5"}]
}If we save the preceding JSON to a file called jsondata.json:
import json from pprint import pprint with open('jsondata.json') as file: data = json.load(file) pprint(data)
Natural language processing is one of the more difficult things to do in machine learning because it is focuses on what machines, at the moment, are not very good at: understanding the structure in complex phenomena.
As a starting point, we can make a few statements about the problem space we are considering. The number of words in any language is usually very large compared to the subset of words that are used in a particular conversation. Our data is sparse compared to the space it exists in. Moreover, words tend to appear in predefined sequences. Certain words are more likely to appear together. Sentences have a certain structure. Different social settings, such as at work, home, or out socializing; or in formal settings such as communicating with regulatory authorities, government, and bureaucratic settings, all require the use overlapping subsets of a vocabulary. A part from cues such as body language, intonation eye contact, and so forth, the social setting is probably the most important factor when trying to extract meaning from natural language.
To work with natural language in Python, we can use the the Natural Language Tool Kit (NLTK). If it is not installed, you can execute the pip install -U nltk command.
The NLTK also comes with a large library of lexical resources. You will need to download these separately, and NLTK has a download manager accessible through the following code:
import nltk nltk.download()
A window should open where you can browse through the various files. This includes a range of books and other written material, as well as various lexical models. To get started, you can just download the package, Book.
A text corpus is a large body of text consisting of numerous individual text files. NLTK comes with corpora from a variety of sources such as classical literature (the Gutenberg Corpus), the web and chat text, Reuter news, and corpus containing text categorized by genres such as new, editorial, religion, fiction, and so on. You can also load any collection of text files using the following code:
from nltk.corpus import PlaintextCorpusReader corpusRoot= 'path/to/corpus' yourCorpus=PlaintextCorpusReader(corpusRoot, '.*')
The second argument to the PlaintextCorpusReader method is a regular expression indicating the files to include. Here, it simply indicates that all the files in that directory are included. This second parameter could also be a list of file locations, such as ['file1', 'dir2/file2'].
Let's take a look at one of the existing corpora, and as an example, we are going to load the Brown corpus:
from nltk.corpus import brown cat=brown.categories() print(cat) ['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
The Brown corpus is useful because it enables us to study the systemic differences between genres. Here is an example:
from nltk.corpus import brown cats=brown.categories() for cat in cats: text=brown.words(categories=cat) fdist = nltk.FreqDist(w.lower() for w in text) posmod = ['love', 'happy', 'good', 'clean'] negmod = ['hate', 'sad', 'bad', 'dirty'] pcount=[] ncount=[] for m in posmod: pcount.append(fdist[m]) for m in negmod: ncount.append(fdist[m]) print(cat + ' positive: ' + str(sum(pcount))) print(cat + ' negative: ' + str(sum(ncount))) rat=sum(pcount)/sum(ncount) print('ratio= %s'%rat ) print()
Here, we have sort of extracted sentiment data from different genres by comparing the occurrences of four positive sentiment words with their antonyms.
Images are a rich and easily available source of data, and they are useful for learning applications such as object recognition, grouping, grading objects, as well as image enhancement. Images, of course, can be put together as a time series. Animating images is useful for both presentation and analysis; for example, we can use video to study trajectories, monitor environments, and learn dynamic behavior.
Image data is structured as a grid or matrix with color values assigned to each pixel. We can get a feel of how this works by using the Python Image Library. For this example, you will need to execute the following lines:
from PIL import Image from matplotlib import pyplot as plt import numpy as np image= np.array(Image.open('data/sampleImage.jpg')) plt.imshow(image, interpolation='nearest') plt.show() print(image.shape) Out[10]: (536, 800, 3)
We can see that this particular image is 536 pixels wide and 800 pixels high. There are 3 values per pixel, representing color values between 0 and 255, for red, green, and blue respectively. Note that the co-ordinate system's origin (0,0) is the top left corner. Once we have our images as NumPy arrays, we can start working with them in interesting ways, for example, taking slices:
im2=image[0:100,0:100,2]
Many social networking platforms have Application programming interfaces (APIs) that give the programmer access to various features. These interfaces can generate quite large amounts of streaming data. Many of these APIs have variable support for Python 3 and some other operating systems, so be prepared to do some research regarding the compatibility of systems.
Gaining access to a platform's API usually involves registering an application with the vendor and then using supplied security credentials, such as public and private keys, to authenticate your application.
Let's take a look at the Twitter API, which is relatively easy to access and has a well-developed library for Python. To get started, we need to load the Twitter library. If you do not have it already, simply execute the pip install twitter command from your Python command prompt.
You will need a Twitter account. Sign in and go to apps.twitter.com. Click on the Create New App button and fill out the details on the Create An Application page. Once you have submitted this, you can access your credential information by clicking on your app from the application management page and then clicking on the Keys and Access Tokens tab.
The four items we are interested in here are the API Key, the API Secret, The Access token, and the Access Token secret. Now, to create our Twitter object:
from twitter import Twitter, OAuth #create our twitter object t = Twitter(auth=OAuth(accesToken, secretToken, apiKey, apiSecret)) #get our home time line home=t.statuses.home_timeline() #get a public timeline anyone= t.statuses.user_timeline(screen_name="abc730") #search for a hash tag pycon=t.search.tweets(q="#pycon") #The screen name of the user who wrote the first 'tweet' user=anyone[0]['user']['screen_name'] #time tweet was created created=anyone[0]['created_at'] #the text of the tweet text= anyone[0]['text']
You will, of course, need to fill in the authorization credentials that you obtained from Twitter earlier. Remember that in a publicly accessible application, you never have these credentials in a human-readable form, and certainly not in the file itself, and preferably encrypted outside a public directory.