Outliers are a common issue in data analysis. Although an exact definition of outliers doesn't exist, we know that outliers can influence means and regression results. Outliers are values that are anomalous. Usually, outliers are caused by a measurement error, but the outliers are sometimes real. In the second case, we may be dealing with two or more types of data related to different phenomena.

The data for this recipe is described at https://vincentarelbundock.github.io/Rdatasets/doc/robustbase/starsCYG.html (retrieved August 2015). It consists of logarithmic effective temperature and logarithmic light intensity for 47 stars in a certain star cluster. Any astronomers reading this paragraph will know the Hertzsprung-Russell diagram. In data analysis terms, the diagram is a scatter plot, but for astronomers, it is of course more than that. The Hertzsprung Russell diagram was defined around 1910 and features a diagonal line (not entirely straight) called the main sequence. Most stars in our data set should be on the main sequence with four outliers in the upper-left corner. These outliers are classified as giants.

We have many strategies to deal with outliers. In this recipe, we will use the two simplest strategies: clipping with the NumPy clip() function and completely removing the outliers. For this example, I define outliers as values 1.5 interquartile ranges removed from the box defined by the 1st and 3rd quartile.

How to do it...

The following steps show how to clip and filter outliers:

1. The imports are as follows:

   import statsmodels.api as sm
   import matplotlib.pyplot as plt
   import numpy as np
   import seaborn as sns
   import dautil as dl
   from IPython.display import HTML

2. Define the following function to filter outliers:

   def filter_outliers(a):
       b = a.copy()
       bmin, bmax = dl.stats.outliers(b)
       b[bmin > b] = np.nan
       b[b > bmax] = np.nan
   
       return b

3. Load and clip outliers as follows:

   starsCYG = sm.datasets.get_rdataset("starsCYG", "robustbase", cache=True).data
   
   clipped = starsCYG.apply(dl.stats.clip_outliers)

4. Filter outliers as follows:

   filtered = starsCYG.copy()
   filtered['log.Te'] = filter_outliers(filtered['log.Te'].values)
   filtered['log.light'] = filter_outliers(filtered['log.light'].values)
   filtered.dropna()

5. Plot the result with the following code:

   sp = dl.plotting.Subplotter(3, 1, context)
   sp.label()
   sns.regplot(x='log.Te', y='log.light', data=starsCYG, ax=sp.ax)
   sp.label(advance=True)
   sns.regplot(x='log.Te', y='log.light', data=clipped, ax=sp.ax)
   sp.label(advance=True)
   sns.regplot(x='log.Te', y='log.light', data=filtered, ax=sp.ax)
   plt.tight_layout()
   HTML(dl.report.HTMLBuilder().watermark())

Refer to the following screenshot for the end result (refer to the outliers.ipynb file in this book's code bundle):