Kernel Smoother - Local Linear Regression

Local Linear Regression

In the two previous sections we assumed that the underlying Y(X) function is locally constant, therefore we were able to use the weighted average for the estimation. The idea of local linear regression is to fit locally a straight line (or a hyperplane for higher dimensions), and not the constant (horizontal line). After fitting the line, the estimation is provided by the value of this line at X0 point. By repeating this procedure for each X0, one can get the estimation function . Like in previous section, the window width is constant Formally, the local linear regression is computed by solving a weighted least square problem.

For one dimension (p = 1):

\begin{align} & \min_{\alpha (X_0),\beta (X_0)} \sum\limits_{i=1}^N {K_{h_{\lambda }}(X_0,X_i)\left( Y(X_i)-\alpha (X_0)-\beta (X_{0})X_i \right)^2} \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\Downarrow \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\hat{Y}(X_{0})=\alpha (X_{0})+\beta (X_{0})X_{0} \\ \end{align}

The closed form solution is given by:

where:

  • B^{T}=\left( \begin{matrix} 1 & 1 & \dots & 1 \\ X_{1} & X_{2} & \dots & X_{N} \\ \end{matrix} \right)

Example:

The resulting function is smooth, and the problem with the biased boundary points is solved.

Local linear regression can be applied to any dimensional space, though the question of what is a local neighborhood becomes more complicated. It is common to use k nearest training points to a test point to fit the local linear regression. This can lead to high variance of the fitted function. To bound the variance, the set of training points should contain the test point in their convex hull (see Gupta et al. reference).

Read more about this topic:  Kernel Smoother

Famous quotes containing the word local:

    “Wags try to invent new stories to tell about the legislature, and end by telling the old one about the senator who explained his unaccustomed possession of a large roll of bills by saying that someone pushed it over the transom while he slept. The expression “It came over the transom,” to explain any unusual good fortune, is part of local folklore.”
    —For the State of Montana, U.S. public relief program (1935-1943)