Gradient Descent - Solution of A Non-linear System

Solution of A Non-linear System

Gradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, x₁, x₂, and x₃. This example shows one iteration of the gradient descent.

Consider a nonlinear system of equations:

$\begin{cases} 3x_1-\cos(x_2x_3)-\tfrac{3}{2}=0 \\ 4x_1^2-625x_2^2+2x_2-1=0 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3}=0 \end{cases}$

suppose we have the function

$G(\mathbf{x}) = \begin{bmatrix} 3x_1-\cos(x_2x_3)-\tfrac{3}{2} \\ 4x_1^2-625x_2^2+2x_2-1 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3} \\ \end{bmatrix}$

where

$\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix}$

and the objective function

With initial guess

$\mathbf{x}^{(0)}=\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} =\begin{bmatrix} 0 \\ 0 \\ 0 \\ \end{bmatrix}$

We know that

where

The Jacobian matrix

$J_G = \begin{bmatrix} 3 & \sin(x_2x_3)x_3 & \sin(x_2x_3)x_2 \\ 8x_1 & -1250x_2+2 & 0 \\ -x_2\exp{(-x_1x_2)} & -x_1\exp(-x_1x_2) & 20\\ \end{bmatrix}$

Then evaluating these terms at

$J_G \left(\mathbf{x}^{(0)}\right) = \begin{bmatrix} 3 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 20 \end{bmatrix}$

and

$G(\mathbf{x}^{(0)}) = \begin{bmatrix} -2.5\\ -1\\ 10.472 \end{bmatrix}$

So that

$\mathbf{x}^{(1)}=0-\gamma_0 \begin{bmatrix} -7.5\\ -2\\ 209.44 \end{bmatrix}.$

and

$F \left(\mathbf{x}^{(0)}\right) = 0.5((-2.5)^2 + (-1)^2 + (10.472)^2) = 58.456$

Now a suitable must be found such that . This can be done with any of a variety of line search algorithms. One might also simply guess which gives

$\mathbf{x}^{(1)}=\begin{bmatrix} 0.0075 \\ 0.002 \\ -0.20944 \\ \end{bmatrix}$

evaluating at this value,

$F \left(\mathbf{x}^{(1)}\right) = 0.5((-2.48)^2 + (-1.00)^2 + (6.28)^2) = 23.306$

The decrease from to the next step's value of is a sizable decrease in the objective function. Further steps would reduce its value until a solution to the system was found.

Read more about this topic: Gradient Descent

Famous quotes containing the words solution of, solution and/or system:

“The truth of the thoughts that are here set forth seems to me unassailable and definitive. I therefore believe myself to have found, on all essential points, the final solution of the problems. And if I am not mistaken in this belief, then the second thing in which the value of this work consists is that it shows how little is achieved when these problems are solved.”
—Ludwig Wittgenstein (1889–1951)

“All the followers of science are fully persuaded that the processes of investigation, if only pushed far enough, will give one certain solution to each question to which they can be applied.... This great law is embodied in the conception of truth and reality. The opinion which is fated to be ultimately agreed to by all who investigate is what we mean by the truth, and the object represented in this opinion is the real.”
—Charles Sanders Peirce (1839–1914)

“Intimately concerned as we are with the system of Europe, it does not follow that we are therefore called upon to mix ourselves on every occasion, with a restless and meddling activity, in the concerns of the nations which surround us.”
—George Canning (1770–1827)