When the target variable is binary, many machine learning algorithms will return the prediction for the observation in the form of a score that ranges from 0 to 1. Therefore, the positive or negative value of the prediction depends on where we set the threshold in that range. For example, if we build a model to predict cancer malignancy and determine that a particular patient's malignancy likelihood is 0.65, choosing a positive threshold of 0.60 makes a positive prediction for that patient, while choosing a threshold of 0.70 makes a negative prediction for that patient. All of the performance scores vary according to where we set the threshold. Certain thresholds will lead to better performance than others, depending on our goal for detection. For example, if we are interested in cancer detection, setting a threshold to a low value such as 0.05 will increase the sensitivity of our model, at the expense of the specificity, but this may be desired because we may not mind the false positives as long as we can identify every patient who is possibly at risk for cancer.
Perhaps the most common performance measurement paradigm for binary-valued outcome variables is to construct a receiver operating characteristic (ROC) curve. In this curve, we plot the values of two measures, the false-positive rate, and the sensitivity, as we vary the threshold from 0 to 1. The sensitivity is usually inversely related to the false-positive rate, yielding a lowercase-r-shaped curve in most cases. The stronger the model, the higher the sensitivity and the lower the false positive rate will generally be, and the area under the curve (AUC) will tend to approach 1. The AUC can, therefore, be used to compare models (for the same use case) while removing the dependency on the threshold's value.
The example ROC plot (Example ROC Curves, 2016) shown as follows has two ROC curves, a dark one and a light one. Because the red (dark) curve has a greater area under the curve than the lighter curve, the model measured by the dark curve can be seen as being better performing than that reflected by the lighter curve:
