Stochastic Spectral Descent for Discrete Graphical Models

David Carlson, Ya Ping Hsieh, Edo Collins, Lawrence Carin, Volkan Cevher

Research output: Contribution to journalArticlepeer-review

19 Scopus citations

Abstract

Interest in deep probabilistic graphical models has increased in recent years, due to their state-of-the-art performance on many machine learning applications. Such models are typically trained with the stochastic gradient method, which can take a significant number of iterations to converge. Since the computational cost of gradient estimation is prohibitive even for modestly sized models, training becomes slow and practically usable models are kept small. In this paper we propose a new, largely tuning-free algorithm to address this problem. Our approach derives novel majorization bounds based on the Schatten-∞ norm. Intriguingly, the minimizers of these bounds can be interpreted as gradient methods in a non-Euclidean space. We thus propose using a stochastic gradient method in non-Euclidean space. We both provide simple conditions under which our algorithm is guaranteed to converge, and demonstrate empirically that our algorithm leads to dramatically faster training and improved predictive ability compared to stochastic gradient descent for both directed and undirected graphical models.
Original languageEnglish (US)
Pages (from-to)296-311
Number of pages16
JournalIEEE Journal on Selected Topics in Signal Processing
Volume10
Issue number2
DOIs
StatePublished - Mar 1 2016
Externally publishedYes

Fingerprint

Dive into the research topics of 'Stochastic Spectral Descent for Discrete Graphical Models'. Together they form a unique fingerprint.

Cite this