Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting

Keyi Chen, John Langford, Francesco Orabona

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance. In practical applications, however, there remains an empirical gap between tuned stochastic gradient descent (SGD) and PFSGD. In this paper, we close the empirical gap with a new parameter-free algorithm based on continuous-time Coin-Betting on truncated models. The new update is derived through the solution of an Ordinary Differential Equation (ODE) and solved in a closed form. We show empirically that this new parameter-free algorithm outperforms algorithms with the “best default” learning rates and almost matches the performance of finely tuned baselines without anything to tune.
Original languageEnglish (US)
Title of host publicationProceedings of the 36th AAAI Conference on Artificial Intelligence, AAAI 2022
PublisherAssociation for the Advancement of Artificial Intelligence
Pages6239-6247
Number of pages9
ISBN (Print)1577358767
StatePublished - Jun 30 2022
Externally publishedYes

Fingerprint

Dive into the research topics of 'Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting'. Together they form a unique fingerprint.

Cite this