Zalan Fabian

Postdoc at USC Center on AI Foundations for Science.

Yue Niu, Zalan Fabian, Sunwoo Lee, Mahdi Soltanolkotabi, Salman Avestimehr
Published in Transactions on Machine Learning Research, 2023

Quasi-Newton methods still face significant challenges in training large-scale neural networks due to additional compute costs in the Hessian related computations and instability issues in stochastic training. A well-known method, L-BFGS that efficiently approximates the Hessian using history parameter and gradient changes, suffers convergence instability in stochastic training. So far, attempts that adapt L-BFGS to large-scale stochastic training incur considerable extra overhead, which offsets its convergence benefits in wall-clock time. In this paper, we propose mL-BFGS, a lightweight momentum-based L-BFGS algorithm that paves the way for quasi-Newton (QN) methods in large-scale distributed deep neural network (DNN) optimization. mL-BFGS introduces a nearly cost-free momentum scheme into L-BFGS update and greatly reduces stochastic noise in the Hessian, therefore stabilizing convergence during stochastic optimization. For model training at a large scale, mL-BFGS approximates a block-wise Hessian, thus enabling distributing compute and memory costs across all computing nodes. We provide a supporting convergence analysis for mL-BFGS in stochastic settings. To investigate mL-BFGS’s potential in large-scale DNN training, we train benchmark neural models using mL-BFGS and compare performance with baselines (SGD, Adam, and other quasi-Newton methods). Results show that mL-BFGS achieves both noticeable iteration-wise and wall-clock speedup.

Paper

Recommended citation

Niu, Y., Fabian Z., Lee, S., Soltanolkotabi, M. and Avestimehr S., 2023, Transactions on Machine Learning Research

BibTeX

@article{niu2023ml,
  title={mL-BFGS: A Momentum-based L-BFGS for Distributed Large-scale Neural Network Optimization},
  author={Niu, Yue and Fabian, Zalan and Lee, Sunwoo and Soltanolkotabi, Mahdi and Avestimehr, Salman},
  journal={Transactions on Machine Learning Research},
  year={2023}
}

Updated: