Descriptive statistics, in the simplest sense, will tell you the basic information about your dataset: how many non-missing observations there are in your dataset, the mean and the standard deviation for the column, as well as the min and max values.

However, first things first—let's load our data and convert it to a Spark DataFrame:

import pyspark.sql.types as typ

First, we load the only module we will need. The pyspark.sql.types exposes all the data types we can use, such as IntegerType() or FloatType().

Next, we read the data in and remove the header line using the .filter(...) method. This is followed by splitting the row on each comma (since this is a .csv file) and converting each element to an integer:

fraud = sc.textFile('ccFraud.csv.gz')
header = fraud.first()

fraud = fraud 
    .filter(lambda row: row != header) 
    .map(lambda row: [int(elem) for elem in row.split(',')])

Next, we create the schema for our DataFrame:

fields = [
    *[
        typ.StructField(h[1:-1], typ.IntegerType(), True)
        for h in header.split(',')
    ]
]
schema = typ.StructType(fields)

Finally, we create our DataFrame:

fraud_df = spark.createDataFrame(fraud, schema)

Having created our fraud_df DataFrame, we can calculate the basic descriptive statistics for our dataset. However, you need to remember that even though all of our features appear as numeric in nature, some of them are categorical (for example, gender or state).

Here's the schema of our DataFrame:

fraud_df.printSchema()

The representation is shown here:

Also, no information would be gained from calculating the mean and standard deviation of the custId column, so we will not be doing that.

For a better understanding of categorical columns, we will count the frequencies of their values using the .groupby(...) method. In this example, we will count the frequencies of the gender column:

fraud_df.groupby('gender').count().show()

The preceding code will produce the following output:

As you can see, we are dealing with a fairly imbalanced dataset. What you would expect to see is an equal distribution for both genders.

For the truly numerical features, we can use the .describe() method:

numerical = ['balance', 'numTrans', 'numIntlTrans']
desc = fraud_df.describe(numerical)
desc.show()

The .show() method will produce the following output:

Even from these relatively few numbers we can tell quite a bit:

Here's how you check the skeweness (we will do it for the 'balance' feature only):

fraud_df.agg({'balance': 'skewness'}).show()

The preceding code produces the following output:

A list of aggregation functions (the names are fairly self-explanatory) includes: avg(), count(), countDistinct(), first(), kurtosis(), max(), mean(), min(), skewness(), stddev(), stddev_pop(), stddev_samp(), sum(), sumDistinct(), var_pop(), var_samp() and variance().

Another highly useful measure of mutual relationships between features is correlation. Your model would normally include only those features that are highly correlated with your target. However, it is almost equally important to check the correlation between the features; including features that are highly correlated among them (that is, are collinear) may lead to unpredictable behavior of your model, or might unnecessarily complicate it.

Calculating correlations in PySpark is very easy once your data is in a DataFrame form. The only difficulties are that the .corr(...) method supports the Pearson correlation coefficient at the moment, and it can only calculate pairwise correlations, such as the following:

fraud_df.corr('balance', 'numTrans')

In order to create a correlations matrix, you can use the following script:

n_numerical = len(numerical)

corr = []

for i in range(0, n_numerical):
    temp = [None] * i
    
    for j in range(i, n_numerical):
        temp.append(fraud_df.corr(numerical[i], numerical[j]))
    corr.append(temp)

The preceding code will create the following output:

As you can see, the correlations between the numerical features in the credit card fraud dataset are pretty much non-existent. Thus, all these features can be used in our models, should they turn out to be statistically sound in explaining our target.

Having checked the correlations, we can now move on to visually inspecting our data.