Measuring A Classifier With Sensitivity and Specificity
Suppose you are training your own classifier, and you wish to measure its performance using the well-accepted metrics of sensitivity and specificity. It may be instructive to compare your classifier to a random classifier that flips a coin based on the prevalence of a disease. Suppose that the probability a person has the disease is and the probability that they do not is . Suppose then that we have a random classifier that guesses that you have the disease with that same probability and guesses you do not with the same probability .
The probability of a true positive is the probability that you have the disease and the random classifier guesses that you do, or . With similar reasoning, the probability of a false negative is . From the definitions above, the sensitivity of this classifier is . With more similar reasoning, we can calculate the specificity as .
So, while the measure itself is independent of disease prevalence, the performance of this random classifier depends on disease prevalence. Your classifier may have performance that is like this random classifier, but with a better-weighted coin (higher sensitivity and specificity). So, these measures may be influenced by disease prevalence. An alternative measure of performance is the Matthews correlation coefficient, for which any random classifier will get an average score of 0.
The extension of this concept to non-binary classifications yields the confusion matrix.
Read more about this topic: Classification Rule
Famous quotes containing the words measuring a, measuring and/or sensitivity:
“As an example of just how useless these philosophers are for any practice in life there is Socrates himself, the one and only wise man, according to the Delphic Oracle. Whenever he tried to do anything in public he had to break off amid general laughter. While he was philosophizing about clouds and ideas, measuring a flea’s foot and marveling at a midge’s humming, he learned nothing about the affairs of ordinary life.”
—Desiderius Erasmus (c. 1466–1536)
“As an example of just how useless these philosophers are for any practice in life there is Socrates himself, the one and only wise man, according to the Delphic Oracle. Whenever he tried to do anything in public he had to break off amid general laughter. While he was philosophizing about clouds and ideas, measuring a flea’s foot and marveling at a midge’s humming, he learned nothing about the affairs of ordinary life.”
—Desiderius Erasmus (c. 1466–1536)
“What we think of as our sensitivity is only the higher evolution of terror in a poor dumb beast. We suffer for nothing. Our own death wish is our only real tragedy.”
—Mario Puzo (b. 1920)