Using the super-learner model

The usage of the model is easy-we need to provide a dataset with a single column called reviewText and transform it with superLearnerModel:

val review = "Although I love this movie, I can barely watch it, it is so real....."
val reviewToScore= sc.parallelize(Seq(review)).toDF("reviewText")
val reviewPrediction= superLearnerModel.transform(reviewToScore)

The returned prediction reviewPrediction is a dataset with the following structure:

reviewPrediction.printSchema()

The first column contains the predicted value, which was decided based on the F1 threshold. The columns p0 and p1 represent probabilities of individual prediction classes.

If we explore the content of the returned dataset, it contains a single row:

reviewPrediction.show()
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset