Modular deep belief networks that do not forget

Leo Pape, Faustino Gomez, Mark Ring, Jurgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations


Deep belief networks (DBNs) are popular for learning compact representations of high-dimensional data. However, most approaches so far rely on having a single, complete training set. If the distribution of relevant features changes during subsequent training stages, the features learned in earlier stages are gradually forgotten. Often it is desirable for learning algorithms to retain what they have previously learned, even if the input distribution temporarily changes. This paper introduces the M-DBN, an unsupervised modular DBN that addresses the forgetting problem. M-DBNs are composed of a number of modules that are trained only on samples they best reconstruct. While modularization by itself does not prevent forgetting, the M-DBN additionally uses a learning method that adjusts each module's learning rate proportionally to the fraction of best reconstructed samples. On the MNIST handwritten digit dataset module specialization largely corresponds to the digits discerned by humans. Furthermore, in several learning tasks with changing MNIST digits, M-DBNs retain learned features even after those features are removed from the training data, while monolithic DBNs of comparable size forget feature mappings learned before. © 2011 IEEE.
Original languageEnglish (US)
Title of host publicationProceedings of the International Joint Conference on Neural Networks
Number of pages8
StatePublished - Oct 24 2011
Externally publishedYes


Dive into the research topics of 'Modular deep belief networks that do not forget'. Together they form a unique fingerprint.

Cite this