Measuring A Classifier With Sensitivity and Specificity
Suppose you are training your own classifier, and you wish to measure its performance using the well-accepted metrics of sensitivity and specificity. It may be instructive to compare your classifier to a random classifier that flips a coin based on the prevalence of a disease. Suppose that the probability a person has the disease is and the probability that they do not is . Suppose then that we have a random classifier that guesses that you have the disease with that same probability and guesses you do not with the same probability .
The probability of a true positive is the probability that you have the disease and the random classifier guesses that you do, or . With similar reasoning, the probability of a false negative is . From the definitions above, the sensitivity of this classifier is . With more similar reasoning, we can calculate the specificity as .
So, while the measure itself is independent of disease prevalence, the performance of this random classifier depends on disease prevalence. Your classifier may have performance that is like this random classifier, but with a better-weighted coin (higher sensitivity and specificity). So, these measures may be influenced by disease prevalence. An alternative measure of performance is the Matthews correlation coefficient, for which any random classifier will get an average score of 0.
Read more about this topic: Binary Classification
Famous quotes containing the words measuring a, measuring and/or sensitivity:
“As an example of just how useless these philosophers are for any practice in life there is Socrates himself, the one and only wise man, according to the Delphic Oracle. Whenever he tried to do anything in public he had to break off amid general laughter. While he was philosophizing about clouds and ideas, measuring a flea’s foot and marveling at a midge’s humming, he learned nothing about the affairs of ordinary life.”
—Desiderius Erasmus (c. 1466–1536)
“Man always made, and still makes, grotesque blunders in selecting and measuring forces, taken at random from the heap, but he never made a mistake in the value he set on the whole, which he symbolized as unity and worshipped as God. To this day, his attitude towards it has never changed, though science can no longer give to force a name.”
—Henry Brooks Adams (1838–1918)
“The sensitivity of men to small matters, and their indifference to great ones, indicates a strange inversion.”
—Blaise Pascal (1623–1662)