Limited-memory BFGS

Please note that the content of this book primarily consists of articles available from Wikipedia or other free sources online. The limited-memory BFGS algorithm is a member of the broad family of quasi-Newton optimization methods that uses a limited memory variation of the Broyden–Fletcher–Goldfarb–Shanno update to approximate the inverse Hessian matrix. Unlike the original BFGS method which stores a dense n \times n approximation, L-BFGS stores only a few vectors that represent the approximation implicitly. Due to its moderate memory requirement, L-BFGS method is particularly well suited for optimization problems with a large number of variables. L-BFGS never explicitly forms or stores H_k. Instead, it maintains a history of the past m\,\! updates of the position x\,\! and gradient \nabla f(x), where generally the history m\,\! can be short, often less than 10. These updates are used to implicitly do operations requiring the H_k-vector product.