Mallows' Cp - Definition and Properties

Definition and Properties

Mallows' C_p addresses the issue of overfitting, in which model selection statistics such as the residual sum of squares always get smaller as more variables are added to a model. Thus, if we aim to select the model giving the smallest residual sum of squares, the model including all variables would always be selected. The C_p statistic calculated on a sample of data estimates the mean squared prediction error (MSPE) as its population target

where is the fitted value from the regression model for the jth case, E(Y_j | X_j) is the expected value for the jth case, and σ2 is the error variance (assumed constant across the cases). The MSPE will not automatically get smaller as more variables are added. The optimum model under this criterion is a compromise influenced by the sample size, the effect sizes of the different predictors, and the degree of collinearity between them.

If P regressors are selected from a set of K > P, the C_p statistic for that particular set of regressors is defined as :

where

• is the error sum of squares for the model with P regressors,
• Y_pi is the predicted value of the i'th observation of Y from the P regressors,
• S2 is the residual mean square after regression on the complete set of K regressors and can be estimated by mean square error MSE,
• and N is the sample size.