Stepwise Regression - Criticism

Criticism

Stepwise regression procedures are used in data mining, but are controversial. Several points of criticism have been made.

• The tests themselves are biased, since they are based on the same data. (Rencher and Pun, 1980, Copas, 1983). Wilkinson and Dalall (1981) computed percentage points of the multiple correlation coefficient by simulation and showed that a final regression obtained by forward selection, said by the F-procedure to be significant at 0.1% was in fact only significant at 5%.

• When estimating the degrees of freedom, the number of the candidate independent variables from the best fit selected is smaller than the total number of final model variables, causing the fit to appear better than it is when adjusting the r2 value for the number of degrees of freedom. It is important to consider how many degrees of freedom have been used in the entire model, not just count the number of independent variables in the resulting fit.

• Models that are created may be too-small than the real models in the data.

Critics regard the procedure as a paradigmatic example of data dredging, intense computation often being an inadequate substitute for subject area expertise. Additionally, the results of stepwise regression are often used incorrectly without adjusting them for the fact that model selection has occurred. Especially the practice of fitting the final selected model as if no model selection had taken place and reporting of estimates and confidence intervals as if least-squares theory were valid for them, has been described as a scandal. Widespread incorrect usage and the availability of alternatives such as Ensemble learning, leaving all variables in the model or using expert judgement to identify relevant variables have lead to calls to totally avoid stepwise model selection.