Cosine similarity

Word similarity/dissimilarity is measured via cosine similarity, which has a very nice property of being bound between -1 and 1. Perfect similarity between two words will yield a score of 1, no relation will yield 0, and -1 means that they are opposites.

Note that the cosine similarity function for the word2vec algorithm (again, just the CBOW implementation in Spark, for now) is already baked into MLlib, which we will see shortly.

Take a look at the following diagram:

For those who are interested in other measures of similarity, a recent research has been published that makes a strong case for using Earth-Mover's Distance (EMD), which is a different method from cosine similarity, requiring some additional calculation, but shows promising early results.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset