Gradient Descent - Solution of A Non-linear System

Solution of A Non-linear System

Gradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, x₁, x₂, and x₃. This example shows one iteration of the gradient descent.

Consider a nonlinear system of equations:

$\begin{cases} 3x_1-\cos(x_2x_3)-\tfrac{3}{2}=0 \\ 4x_1^2-625x_2^2+2x_2-1=0 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3}=0 \end{cases}$

suppose we have the function

$G(\mathbf{x}) = \begin{bmatrix} 3x_1-\cos(x_2x_3)-\tfrac{3}{2} \\ 4x_1^2-625x_2^2+2x_2-1 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3} \\ \end{bmatrix}$

where

$\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix}$

and the objective function

With initial guess

$\mathbf{x}^{(0)}=\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} =\begin{bmatrix} 0 \\ 0 \\ 0 \\ \end{bmatrix}$

We know that

where

The Jacobian matrix

$J_G = \begin{bmatrix} 3 & \sin(x_2x_3)x_3 & \sin(x_2x_3)x_2 \\ 8x_1 & -1250x_2+2 & 0 \\ -x_2\exp{(-x_1x_2)} & -x_1\exp(-x_1x_2) & 20\\ \end{bmatrix}$

Then evaluating these terms at

$J_G \left(\mathbf{x}^{(0)}\right) = \begin{bmatrix} 3 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 20 \end{bmatrix}$

and

$G(\mathbf{x}^{(0)}) = \begin{bmatrix} -2.5\\ -1\\ 10.472 \end{bmatrix}$

So that

$\mathbf{x}^{(1)}=0-\gamma_0 \begin{bmatrix} -7.5\\ -2\\ 209.44 \end{bmatrix}.$

and

$F \left(\mathbf{x}^{(0)}\right) = 0.5((-2.5)^2 + (-1)^2 + (10.472)^2) = 58.456$

Now a suitable must be found such that . This can be done with any of a variety of line search algorithms. One might also simply guess which gives

$\mathbf{x}^{(1)}=\begin{bmatrix} 0.0075 \\ 0.002 \\ -0.20944 \\ \end{bmatrix}$

evaluating at this value,

$F \left(\mathbf{x}^{(1)}\right) = 0.5((-2.48)^2 + (-1.00)^2 + (6.28)^2) = 23.306$

The decrease from to the next step's value of is a sizable decrease in the objective function. Further steps would reduce its value until a solution to the system was found.

Read more about this topic: Gradient Descent

Famous quotes containing the words solution of, solution and/or system:

“The Settlement ... is an experimental effort to aid in the solution of the social and industrial problems which are engendered by the modern conditions of life in a great city. It insists that these problems are not confined to any one portion of the city. It is an attempt to relieve, at the same time, the overaccumulation at one end of society and the destitution at the other ...”
—Jane Addams (1860–1935)

“To the questions of the officiously meddling police Falter replied absently and tersely; but, when he finally grew tired of this pestering, he pointed out that, having accidentally solved “the riddle of the universe,” he had yielded to artful exhortation and shared that solution with his inquisitive interlocutor, whereupon the latter had died of astonishment.”
—Vladimir Nabokov (1899–1977)

“Television is an excellent system when one has nothing to lose, as is the case with a nomadic and rootless country like the United States, but in Europe the affect of television is that of a bulldozer which reduces culture to the lowest possible denominator.”
—Marc Fumaroli (b. 1932)