ADMB - History and Background - Implementation

Implementation

Work by David Fournier in the 1970s on development of highly parameterized integrated statistical models in fisheries motivated the development of the AUTODIF Library, and ultimately ADMB. The likelihood equations in these models are typically non-linear and estimates of the parameters are obtained by numerical methods.

Early in Fournier's work, it became clear that general numerical solutions to these likelihood problems could only be reliably achieved using function minimization algorithms that incorporate accurate information about the gradients of the likelihood surface. Computing the gradients (i.e. partial derivatives of the likelihood with respect to all model variables) must also be done with the same accuracy as the likelihood computation itself.

Fournier developed a protocol for writing code to compute the required derivatives based on the chain rule of differential calculus. This protocol is very similar to the suite of methods that came to be known as ``reverse mode automatic differentiation .

The statistical models using these methods typically included eight constituent code segments:

1. the objective function;
2. adjoint code to compute the partial derivatives of the objective function with respect to the parameters to be estimated;
3. dedicated memory to contain intermediate data for derivative computations, known as the "gradient stack", and the software to manage it;
4. a function minimizer;
5. an algorithm to check that the derivatives are correct with respect to finite difference approximations;
6. an algorithm to insert model parameters into a vector that can be manipulated by the function minimizer and the corresponding derivative code;
7. an algorithm to return the parameter values to the likelihood computation and the corresponding derivative code; and
8. an algorithm to compute the second partial derivatives of the objective unction with respect to the parameters to be estimated, the Hessian matrix.

Model developers are usually only interested in the first of these constituents. Any programming tools that can reduce the overhead of developing and maintaining the other seven will greatly increase their productivity.

Bjarne Stroustrup began development of C++ in the 1970s at Bell Labs as an enhancement to the C programming language. C++ spread widely, and by 1989, C++ compilers were available for personal computers. The polymorphism of C++ makes it possible to envisage a programming system in which all mathematical operators and functions can be overloaded to automatically compute the derivative contributions of every differentiable numerical operation in any computer program.