An independent variable is denoted as X. If we have multiple independent variables, then you will see each independent variable denoted as X₁, X₂, X₃, …, X_m. A dependent variable is denoted as Y. The X and Y variables represent a dataset of n values (or observations). The mean of X is known as X-bar. If we wish to take every value of X and subtract X-bar from it, we will write it in the following way:

The term average (which is the result of a sum of a listing of values divided by the number of values) also goes by the term mean as well as expectation. Sometimes, we will see the preceding formula written as follows:

You can think of E as a function that computes the average of a list of values.

The result of this operation will be a new dataset with the average being 0. If a value in this dataset is positive, we know that the result is above average. Likewise, negative values will give the result as below average.

We would like to know the spread of this variable. To do this, we will take each value in the dataset, subtract the mean, and then square the result. This produces a dataset that consists of positive values. After this step, add up all of the values and divide the sum by the number of observations to get the average squared distance from the average. This, as we discussed in Chapter 5, Hypothesis Testing, is the population variance. To find out the sample variance, rather than dividing by n, we divide it by (n-1):

We can write the same formula with the help of a cleaner notation by using the E function:

To find out the population standard deviation, we will take the square root of the population variance. The standard deviation is signified by the Greek letter, sigma (denoted as σ):

Now that we know the average distance from the average of the values in the dataset, we can use this to normalize the X variable. To do this, we will divide each value in X- X-bar by the standard deviation:

This normalized version of our dataset still contains positive and negative values, but it is also a measure of how extreme in distance the normalized variable is from the mean. A score between -1 and 1 means that a value is closer to the mean than the typical data value. Scores ranging from -2 to -1 and from 1 to 2 mean that the value is between one and two times distant from the mean than the typical value. Most of your data should fall between a score of -2 and 2. Scores from -3 to -2 and from 2 to 3 indicate that the value is a little more distant. A value greater than 3 or less than -3 means that this value is more than 3 times the distance from the average of the values in the dataset than the typical value. Values with a score in this range are considered rare and indicate special circumstances that merit investigation. Values that deviate significantly from the majority of a dataset are called outliers. When an outlier value has been identified, it possibly represents special circumstances and should be investigated for unique qualities. It can also mean something less special—a noisy data point was not properly identified in the cleaning phase of data analysis.

When working with two variables (in our case, an input and an output variable), we may want to study how the variables move in conjunction with each other. Like variance, the tool known as covariance helps us to measure how variables relate to each other. Instead of one X variable, we now have two—X and Y.

We will begin by subtracting the mean of X from each value of X, whose answer is then multiplied by the result of the mean of Y subtracted from each value of Y:

Again, we add each of these distance measures and divide the sum by the number of observations to find out the population covariance coefficient. To find the sample covariance coefficient, we change n to n-1:

Again, we can write the same formula by using the E notation:

The covariance coefficient is a measurement of how the variables relate to each other. If the X variable increases as the Y variable increases, then this coefficient will have a positive value. Likewise, if X or Y increases and the other variable decreases, then this coefficient will have a negative value.

We can normalize this value as we did before. This normalized value will always have a value from -1 to 1. We need the standard deviations of X and Y (σx and σy). This normalized version of the covariance value is known as the Pearson r correlation coefficient. The formula for this can be denoted as follows:

As with the correlation coefficient, a positive r value informs us that the variables are linearly correlated (as the value of one variable increases, the value of the other increases) and a negative r value informs us that the variables are inversely correlated (as the value of a variable increases, the other decreases). The closer an r value is to the extremes of -1 or 1, the more the strength of the correlation it indicates. An r value that is close to 0 tells us that the connection between the two variables is weak.

The r value is tweaked further to r² (we simply multiply r by itself). Because r ranges from -1 to 1, the value of r² will always be a value from 0 to 1. A higher r² implies that there is stronger evidence of a correlation. A lower r² implies little or no correlation. An r² that is greater than 0.9 is considered to be an excellent correlation, while an r² that is less than 0.5 is considered to be weak. Interpreting an r² is a form of art—it is what you make of it. The r² values go by the name, coefficient of determination.

We will not continue this discussion without talking about a key pitfall of r²—the discovery of an input and output variable that produces a high r² value does not automatically imply that the input has an impact on the output. This is known by the saying, correlation does not imply causation. When data mining for correlations, you are bound to find pairs of variables that produce high r² values. Some of these input variables might have a causal effect on the output variable, and at times, you may find a similar pattern between two unrelated variables. This frequently happens when two output variables are compared. This measurement only tells you what correlates, not whether the discovered correlations are sensible. You might discover a link between the sale of food and the sale of beverages at a restaurant, but there is nothing interesting about this correlation because these variables are dependent on the number of customers who frequent the restaurant.

We can express these formulas to Haskell in the following way:

module LearningDataAnalysis06 where

  import Data.List
  import Graphics.EasyPlot
  import LearningDataAnalysis02
  import LearningDataAnalysis04
  import LearningDataAnalysis05

  {- Covariance -}
  covariance :: [Double] -> [Double] -> Double
  covariance x y = average $ zipWith (xi yi -> (xi-xavg) * (yi- yavg)) x y
    where
      xavg = average x
      yavg = average y

  {- Pearson r Correlation Coefficient -}
  pearsonR :: [Double] -> [Double] -> Double
  pearsonR x y = r
    where
      xstdev = standardDeviation x
      ystdev = standardDeviation y
      r = covariance x y / (xstdev * ystdev)

  {- Pearson r-squared -}
  pearsonRsqrd :: [Double] -> [Double] -> Double
  pearsonRsqrd x y = pearsonR x y ^ 2