Criticism
Stepwise regression procedures are used in data mining, but are controversial. Several points of criticism have been made.
- The tests themselves are biased, since they are based on the same data. (Rencher and Pun, 1980, Copas, 1983). Wilkinson and Dalall (1981) computed percentage points of the multiple correlation coefficient by simulation and showed that a final regression obtained by forward selection, said by the F-procedure to be significant at 0.1% was in fact only significant at 5%.
- When estimating the degrees of freedom, the number of the candidate independent variables from the best fit selected is smaller than the total number of final model variables, causing the fit to appear better than it is when adjusting the r2 value for the number of degrees of freedom. It is important to consider how many degrees of freedom have been used in the entire model, not just count the number of independent variables in the resulting fit.
- Models that are created may be too-small than the real models in the data.
Critics regard the procedure as a paradigmatic example of data dredging, intense computation often being an inadequate substitute for subject area expertise. Additionally, the results of stepwise regression are often used incorrectly without adjusting them for the fact that model selection has occurred. Especially the practice of fitting the final selected model as if no model selection had taken place and reporting of estimates and confidence intervals as if least-squares theory were valid for them, has been described as a scandal. Widespread incorrect usage and the availability of alternatives such as Ensemble learning, leaving all variables in the model or using expert judgement to identify relevant variables have lead to calls to totally avoid stepwise model selection.
Read more about this topic: Stepwise Regression
Famous quotes containing the word criticism:
“A bad short story or novel or poem leaves one comparatively calm because it does not exist, unless it gets a fake prestige through being mistaken for good work. It is essentially negative, it is something that has not come through. But over bad criticism one has a sense of real calamity.”
—Rebecca West (1892–1983)
“Unless criticism refuses to take itself quite so seriously or at least to permit its readers not to, it will inevitably continue to reflect the finicky canons of the genteel tradition and the depressing pieties of the Culture Religion of Modernism.”
—Leslie Fiedler (b. 1917)
“Parents sometimes feel that if they don’t criticize their child, their child will never learn. Criticism doesn’t make people want to change; it makes them defensive.”
—Laurence Steinberg (20th century)