BFGS Method - Algorithm

Algorithm

From an initial guess and an approximate Hessian matrix the following steps are repeated until converges to the solution.

1. Obtain a direction by solving:
2. Perform a line search to find an acceptable stepsize in the direction found in the first step, then update
3. Set

denotes the objective function to be minimized. Convergence can be checked by observing the norm of the gradient, . Practically, can be initialized with, so that the first step will be equivalent to a gradient descent, but further steps are more and more refined by, the approximation to the Hessian.

The first step of the algorithm is carried out using the inverse of the matrix, which is usually obtained efficiently by applying the Sherman–Morrison formula to the fifth line of the algorithm, giving

In statistical estimation problems (such as maximum likelihood or Bayesian inference), credible intervals or confidence intervals for the solution can be estimated from the inverse of the final Hessian matrix. However, these quantities are technically defined by the true Hessian matrix, and the BFGS approximation may not converge to the true Hessian matrix.