Regression Toward The Mean - Definition For Simple Linear Regression of Data Points

Definition For Simple Linear Regression of Data Points

This is the definition of regression toward the mean that closely follows Sir Francis Galton's original usage.

Suppose there are n data points {yi, xi}, where i = 1, 2, …, n. We want to find the equation of the regression line, i.e. the straight line

which would provide a “best” fit for the data points. (Note that a straight line may not be the appropriate regression curve for the given data points.) Here the “best” will be understood as in the least-squares approach: such a line that minimizes the sum of squared residuals of the linear regression model. In other words, numbers α and β solve the following minimization problem:

Find, where Q(\alpha,\beta) = \sum_{i=1}^n\hat{\varepsilon}_i^{\,2} = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2\

Using simple calculus it can be shown that the values of α and β that minimize the objective function Q are

\begin{align} & \hat\beta = \frac{ \sum_{i=1}^{n} (x_{i}-\bar{x})(y_{i}-\bar{y}) }{ \sum_{i=1}^{n} (x_{i}-\bar{x})^2 } = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } = \frac{ \operatorname{Cov} }{ \operatorname{Var} } = r_{xy} \frac{s_y}{s_x}, \\ & \hat\alpha = \bar{y} - \hat\beta\,\bar{x}, \end{align}

where rxy is the sample correlation coefficient between x and y, sx is the standard deviation of x, and sy is correspondingly the standard deviation of y. Horizontal bar over a variable means the sample average of that variable. For example:

Substituting the above expressions for and into yields fitted values

which yields

This shows the role rxy plays in the regression line of standardized data points.

If −1 < rxy < 1, then we say that the data points exhibit regression toward the mean. In other words, if linear regression is the appropriate model for a set of data points whose sample correlation coefficient is not perfect, then there is regression toward the mean. The predicted (or fitted) standardized value of y is closer to its mean than the standardized value of x is to its mean.

Read more about this topic:  Regression Toward The Mean

Famous quotes containing the words definition, simple, data and/or points:

    “The man who knows governments most completely is he who troubles himself least about a definition which shall give their essence. Enjoying an intimate acquaintance with all their particularities in turn, he would naturally regard an abstract conception in which these were unified as a thing more misleading than enlightening.”
    —William James (1842–1910)

    “To whatsoever upright mind, to whatsoever beating heart I speak, to you it is committed to educate men. By simple living, by an illimitable soul, you inspire, you correct, you instruct, you raise, you embellish all. By your own act you teach the beholder how to do the practicable. According to the depth from which you draw your life, such is the depth not only of your strenuous effort, but of your manners and presence.”
    —Ralph Waldo Emerson (1803–1882)

    “To write it, it took three months; to conceive it three minutes; to collect the data in it—all my life.”
    —F. Scott Fitzgerald (1896–1940)

    “the
    Decapitated exclamation points in that Other Woman’s eyes.”
    —Gwendolyn Brooks (b. 1917)