Model Accuracy
A way to test for errors in models created by step-wise regression is to not rely on the model's F-statistic, significance, or multiple-r, but instead assess the model against a set of data that was not used to create the model. The class of techniques is called cross-validation.
Accuracy is measured as correctly classified records in the holdout sample. There are four possible classifications:
- prediction of 0 when the holdout sample has a 0 (True Negative/TN)
- prediction of 0 when the holdout sample has a 1 (False Negative/FN)
- prediction of 1 when the holdout sample has a 0 (False Positive/FP)
- prediction of 1 when the holdout sample has a 1 (True Positive/TP)
These classifications are used to measure Precision and Recall:
The percent of correctly classified observations in the holdout sample is referred to the assessed model accuracy. Additional accuracy can be expressed as the model's ability to correctly classify 0, or the ability to correctly classify 1 in the holdout dataset. The holdout model assessment method is particularly valuable when data are collected in different settings (e.g., at different times or places) or when models are assumed to be generalizable.
Read more about this topic: Logistic Regression
Famous quotes containing the words model and/or accuracy:
“...that absolutely everything beloved and cherished of the bourgeoisie, the conservative, the cowardly, and the impotent—the State, family life, secular art and science—was consciously or unconsciously hostile to the religious idea, to the Church, whose innate tendency and permanent aim was the dissolution of all existing worldly orders, and the reconstitution of society after the model of the ideal, the communistic City of God.”
—Thomas Mann (1875–1955)
“The child who has been taught to make an accurate elevation, plan, and section of a pint pot has had an admirable training in accuracy of eye and hand.”
—Thomas Henry Huxley (1825–95)