Mallows' Cp - Practical Use

Practical Use

The C_p statistic is often used as a stopping rule for various forms of stepwise regression. Mallows proposed the statistic as a criterion for selecting among many alternative subset regressions. Under a model not suffering from appreciable lack of fit (bias), C_p has expectation nearly equal to P; otherwise the expectation is roughly P plus a positive bias term. Nevertheless, even though it has expectation greater than or equal to P, there is nothing to prevent C_p < P or even C_p < 0 in extreme cases. It is suggested that one should choose a subset that has C_p approaching P, from above, for a list of subsets ordered by increasing P. In practice, the positive bias can be adjusted for by selecting a model from the ordered list of subsets, such that C_p < 2P.

Since the sample-based C_p statistic is an estimate of the MSPE, using C_p for model selection does not completely guard against overfitting. For instance, it is possible that the selected model will be one in which the sample C_p was a particularly severe underestimate of the MSPE.

Model selection statistics such as C_p are generally not used blindly, but rather information about the field of application, the intended use of the model, and any known biases in the data are taken into account in the process of model selection.