Simple Linear Regression - Fitting The Regression Line

Fitting The Regression Line

Suppose there are n data points {yi, xi}, where i = 1, 2, …, n. The goal is to find the equation of the straight line

which would provide a "best" fit for the data points. Here the "best" will be understood as in the least-squares approach: such a line that minimizes the sum of squared residuals of the linear regression model. In other words, numbers α and β solve the following minimization problem:

By using either calculus, the geometry of inner product spaces or simply expanding to get a quadratic in α and β, it can be shown that the values of α and β that minimize the objective function Q are

\begin{align} \hat\beta & = \frac{ \sum_{i=1}^{n} (x_{i}-\bar{x})(y_{i}-\bar{y}) }{ \sum_{i=1}^{n} (x_{i}-\bar{x})^2 } = \frac{ \sum_{i=1}^{n}{x_{i}y_{i}} - \frac1n \sum_{i=1}^{n}{x_{i}}\sum_{j=1}^{n}{y_{j}}}{ \sum_{i=1}^{n}({x_{i}^2}) - \frac1n (\sum_{i=1}^{n}{x_{i}})^2 } \\ & = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } = \frac{ \operatorname{Cov} }{ \operatorname{Var} } = r_{xy} \frac{s_y}{s_x}, \\ \hat\alpha & = \bar{y} - \hat\beta\,\bar{x}, \end{align}

where rxy is the sample correlation coefficient between x and y, sx is the standard deviation of x, and sy is correspondingly the standard deviation of y. Horizontal bar over a variable means the sample average of that variable. For example:

Substituting the above expressions for and into

yields

This shows the role plays in the regression line of standardized data points.

Read more about this topic:  Simple Linear Regression

Famous quotes containing the words fitting the, fitting and/or line:

    “Children’s view of the world and their capacity to understand keep expanding as they mature, and they need to ask the same questions over and over, fitting the information into their new level of understanding.”
    —Joanna Cole (20th century)

    “We do not quite say that the new is more valuable because it fits in; but its fitting in is a test of its value—a test, it is true, which can only be slowly and cautiously applied, for we are none of us infallible judges of conformity.”
    —T.S. (Thomas Stearns)

    “A route differs from a road not only because it is solely intended for vehicles, but also because it is merely a line that connects one point with another. A route has no meaning in itself; its meaning derives entirely from the two points that it connects. A road is a tribute to space. Every stretch of road has meaning in itself and invites us to stop. A route is the triumphant devaluation of space, which thanks to it has been reduced to a mere obstacle to human movement and a waste of time.”
    —Milan Kundera (b. 1929)