There's more...

If you do not pass a list of columns to calculate the descriptive statistics over, PySpark will return the statistics for each and every column in your DataFrame. Check out the following snippet:

descriptive_stats_all = no_outliers.describe()
descriptive_stats_all.show()

It will result in the following table:

As you can see, even the string columns got their descriptive statistics which are, however, fairly questionable to interpret.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset