Cross-validation (statistics) - Purpose of Cross Validation

Purpose of Cross Validation

Suppose we have a model with one or more unknown parameters, and a data set to which the model can be fit (the training data set). The fitting process optimizes the model parameters to make the model fit the training data as well as possible. If we then take an independent sample of validation data from the same population as the training data, it will generally turn out that the model does not fit the validation data as well as it fits the training data. This is called overfitting, and is particularly likely to happen when the size of the training data set is small, or when the number of parameters in the model is large. Cross-validation is a way to predict the fit of a model to a hypothetical validation set when an explicit validation set is not available.

Linear regression provides a simple illustration of overfitting. In linear regression we have real response values Y₁, ..., Y_n, and vector covariates X₁, ..., X_p. We can use least squares to fit a hyperplane a + b₁X₁ + ... + b_pX_p between the Y and X data, and then assess the fit using the mean squared error (MSE)

$\sum_i (Y_i - a - b_1X_{1i} - \cdots - b_pX_{pi})^2/n,$

where X_ji is the value of variable X_j corresponding to the ith response value Y_i.

It can be shown under mild assumptions that the expected value of the MSE for the training set is (n − p − 1)/(n + p + 1) < 1 times the expected value of the MSE for the validation set (the expected value is taken over the distribution of training sets). Thus if we fit the model and compute the MSE on the training set, we will get an optimistically biased assessment of how well the model will fit an independent data set. This biased estimate is called the in-sample estimate of the fit, whereas the cross-validation estimate is an out-of-sample estimate.

Since in linear regression it is possible to directly compute the factor (n − p − 1)/(n + p + 1) by which the training MSE underestimates the validation MSE, cross-validation is not practically useful in that setting. However in most other regression procedures (e.g. logistic regression), there is no simple formula to make this adjustment. Cross-validation is a generally applicable way to predict the performance of a model on a validation set using computation in place of mathematical analysis.

Read more about this topic: Cross-validation (statistics)

Famous quotes containing the words purpose of, purpose and/or cross:

“For the purpose of knowledge, one must know how to use that inner current that draws us to a thing, and then the one that, after a time, draws us away from it.”
—Friedrich Nietzsche (1844–1900)

“A major misunderstanding of child rearing has been the idea that meeting a child’s needs is an end in itself, for the purpose of the child’s mental health. Mothers have not understood that this is but one step in social development, the goal of which is to help a child begin to consider others. As a result, they often have not considered their children but have instead allowed their children’s reality to take precedence, out of a fear of damaging them emotionally.”
—Elaine Heffner (20th century)

“Although its growth may seem to have been slow, it is to be remembered that it is not a shrub, or plant, to shoot up in the summer and wither in the frosts. The Red Cross is a part of us—it has come to stay—and like the sturdy oak, its spreading branches shall yet encompass and shelter the relief of the nation.”
—Clara Barton (1821–1912)