Automatic Differentiation - Automatic Differentiation Using Dual Numbers

Automatic Differentiation Using Dual Numbers

Forward mode automatic differentiation is accomplished by augmenting the algebra of real numbers and obtaining a new arithmetic. An additional component is added to every number which will represent the derivative of a function at the number, and all arithmetic operators are extended for the augmented algebra. The augmented algebra is the algebra of dual numbers. Computer programs often implement this using the complex number representation.

Replace every number with the number, where is a real number, but is nothing but a symbol with the property . Using only this, we get for the regular arithmetic

and likewise for subtraction and division.

Now, we may calculate polynomials in this augmented arithmetic. If, then

where denotes the derivative of with respect to its first argument, and, called a seed, can be chosen arbitrarily.

The new arithmetic consists of ordered pairs, elements written, with ordinary arithmetics on the first component, and first order differentiation arithmetic on the second component, as described above. Extending the above results on polynomials to analytic functions we obtain a list of the basic arithmetic and some standard functions for the new arithmetic:

and in general for the primitive function ,

where and are the derivatives of with respect to its first and second arguments, respectively.

When a binary basic arithmetic operation is applied to mixed arguments — the pair and the real number — the real number is first lifted to . The derivative of a function at the point is now found by calculating using the above arithmetic, which gives as the result.