Gradient Descent - Solution of A Non-linear System

Solution of A Non-linear System

Gradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, x1, x2, and x3. This example shows one iteration of the gradient descent.

Consider a nonlinear system of equations:

\begin{cases} 3x_1-\cos(x_2x_3)-\tfrac{3}{2}=0 \\ 4x_1^2-625x_2^2+2x_2-1=0 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3}=0 \end{cases}

suppose we have the function

G(\mathbf{x}) = \begin{bmatrix} 3x_1-\cos(x_2x_3)-\tfrac{3}{2} \\ 4x_1^2-625x_2^2+2x_2-1 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3} \\ \end{bmatrix}

where

\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix}

and the objective function

With initial guess

\mathbf{x}^{(0)}=\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} =\begin{bmatrix} 0 \\ 0 \\ 0 \\ \end{bmatrix}

We know that

where

The Jacobian matrix

J_G = \begin{bmatrix} 3 & \sin(x_2x_3)x_3 & \sin(x_2x_3)x_2 \\ 8x_1 & -1250x_2+2 & 0 \\ -x_2\exp{(-x_1x_2)} & -x_1\exp(-x_1x_2) & 20\\ \end{bmatrix}

Then evaluating these terms at

J_G \left(\mathbf{x}^{(0)}\right) = \begin{bmatrix} 3 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 20 \end{bmatrix}

and

G(\mathbf{x}^{(0)}) = \begin{bmatrix} -2.5\\ -1\\ 10.472 \end{bmatrix}

So that

\mathbf{x}^{(1)}=0-\gamma_0 \begin{bmatrix} -7.5\\ -2\\ 209.44 \end{bmatrix}.

and

F \left(\mathbf{x}^{(0)}\right) = 0.5((-2.5)^2 + (-1)^2 + (10.472)^2) = 58.456

Now a suitable must be found such that . This can be done with any of a variety of line search algorithms. One might also simply guess which gives

\mathbf{x}^{(1)}=\begin{bmatrix} 0.0075 \\ 0.002 \\ -0.20944 \\ \end{bmatrix}

evaluating at this value,

F \left(\mathbf{x}^{(1)}\right) = 0.5((-2.48)^2 + (-1.00)^2 + (6.28)^2) = 23.306

The decrease from to the next step's value of is a sizable decrease in the objective function. Further steps would reduce its value until a solution to the system was found.

Read more about this topic:  Gradient Descent

Famous quotes containing the words solution of, solution and/or system:

    “What is history? Its beginning is that of the centuries of systematic work devoted to the solution of the enigma of death, so that death itself may eventually be overcome. That is why people write symphonies, and why they discover mathematical infinity and electromagnetic waves.”
    —Boris Pasternak (1890–1960)

    “I can’t quite define my aversion to asking questions of strangers. From snatches of family battles which I have heard drifting up from railway stations and street corners, I gather that there are a great many men who share my dislike for it, as well as an equal number of women who ... believe it to be the solution to most of this world’s problems.”
    —Robert Benchley (1889–1945)

    “Delight at having understood a very abstract and obscure system leads most people to believe in the truth of what it demonstrates.”
    —G.C. (Georg Christoph)