Kernel Smoother - Local Linear Regression

Local Linear Regression

In the two previous sections we assumed that the underlying Y(X) function is locally constant, therefore we were able to use the weighted average for the estimation. The idea of local linear regression is to fit locally a straight line (or a hyperplane for higher dimensions), and not the constant (horizontal line). After fitting the line, the estimation is provided by the value of this line at X0 point. By repeating this procedure for each X0, one can get the estimation function . Like in previous section, the window width is constant Formally, the local linear regression is computed by solving a weighted least square problem.

For one dimension (p = 1):

\begin{align} & \min_{\alpha (X_0),\beta (X_0)} \sum\limits_{i=1}^N {K_{h_{\lambda }}(X_0,X_i)\left( Y(X_i)-\alpha (X_0)-\beta (X_{0})X_i \right)^2} \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\Downarrow \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\hat{Y}(X_{0})=\alpha (X_{0})+\beta (X_{0})X_{0} \\ \end{align}

The closed form solution is given by:

where:

  • B^{T}=\left( \begin{matrix} 1 & 1 & \dots & 1 \\ X_{1} & X_{2} & \dots & X_{N} \\ \end{matrix} \right)

Example:

The resulting function is smooth, and the problem with the biased boundary points is solved.

Local linear regression can be applied to any dimensional space, though the question of what is a local neighborhood becomes more complicated. It is common to use k nearest training points to a test point to fit the local linear regression. This can lead to high variance of the fitted function. To bound the variance, the set of training points should contain the test point in their convex hull (see Gupta et al. reference).

Read more about this topic:  Kernel Smoother

Famous quotes containing the word local:

    “Hey, you dress up our town very nicely. You don’t look out the Chamber of Commerce is going to list you in their publicity with the local attractions.”
    —Robert M. Fresco, and Jack Arnold. Dr. Matt Hastings (John Agar)