How it works...

First, we assemble all the features into a single vector using the .VectorAssembler(...) method. Note that we do not use the last column as it is the CoverType feature and this is our target.

Next, we use the .ChiSqSelector(...) method to select the best features based on the pairwise chi-square test between each variable and the target. Based on the values from the test, numTopFeatures, the most predictable features, are selected. The selected vector will contain the top 10 (in this case) most predictable features. The labelCol specifies the target column.

You can learn more about the chi-square test here: http://learntech.uwe.ac.uk/da/Default.aspx?pageid=1440.

Let's check it out:

(
pipeline_sel
.fit(forest)
.transform(forest)
.select(selector.getOutputCol())
.show(5)
)

Here's what you should see from running the preceding snippet:

As you can see, the resulting SparseVector has a length of 10 and includes only the most predictable features. 

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset