Criticism
Stepwise regression procedures are used in data mining, but are controversial. Several points of criticism have been made.
- The tests themselves are biased, since they are based on the same data. (Rencher and Pun, 1980, Copas, 1983). Wilkinson and Dalall (1981) computed percentage points of the multiple correlation coefficient by simulation and showed that a final regression obtained by forward selection, said by the F-procedure to be significant at 0.1% was in fact only significant at 5%.
- When estimating the degrees of freedom, the number of the candidate independent variables from the best fit selected is smaller than the total number of final model variables, causing the fit to appear better than it is when adjusting the r2 value for the number of degrees of freedom. It is important to consider how many degrees of freedom have been used in the entire model, not just count the number of independent variables in the resulting fit.
- Models that are created may be too-small than the real models in the data.
Critics regard the procedure as a paradigmatic example of data dredging, intense computation often being an inadequate substitute for subject area expertise. Additionally, the results of stepwise regression are often used incorrectly without adjusting them for the fact that model selection has occurred. Especially the practice of fitting the final selected model as if no model selection had taken place and reporting of estimates and confidence intervals as if least-squares theory were valid for them, has been described as a scandal. Widespread incorrect usage and the availability of alternatives such as Ensemble learning, leaving all variables in the model or using expert judgement to identify relevant variables have lead to calls to totally avoid stepwise model selection.
Read more about this topic: Stepwise Regression
Famous quotes containing the word criticism:
“However intense my experience, I am conscious of the presence and criticism of a part of me, which, as it were, is not a part of me, but a spectator, sharing no experience, but taking note of it, and that is no more I than it is you. When the play, it may be the tragedy, of life is over, the spectator goes his way. It was a kind of fiction, a work of the imagination only, so far as he was concerned.”
—Henry David Thoreau (1817–1862)
“It is ... pathetic to observe the complete lack of imagination on the part of certain employers and men and women of the upper-income levels, equally devoid of experience, equally glib with their criticism ... directed against workers, labor leaders, and other villains and personal devils who are the objects of their dart-throwing. Who doesn’t know the wealthy woman who fulminates against the “idle” workers who just won’t get out and hunt jobs?”
—Mary Barnett Gilson (1877–?)
“I, with other Americans, have perhaps unduly resented the stream of criticism of American life ... more particularly have I resented the sneers at Main Street. For I have known that in the cottages that lay behind the street rested the strength of our national character.”
—Herbert Hoover (1874–1964)