A Computable Definition of the Spectral Bias

Jonas Kiessling, Filip Thor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Neural networks have a bias towards low frequency functions. This spectral bias has been the subject of several previous studies, both empirical and theoretical. Here we present a computable definition of the spectral bias based on a decomposition of the reconstruction error into a low and a high frequency component. The distinction between low and high frequencies is made in a way that allows for easy interpretation of the spectral bias. Furthermore, we present two methods for estimating the spectral bias. Method 1 relies on the use of the discrete Fourier transform to explicitly estimate the Fourier spectrum of the prediction residual, and Method 2 uses convolution to extract the low frequency components, where the convolution integral is estimated by Monte Carlo methods. The spectral bias depends on the distribution of the data, which is approximated with kernel density estimation when unknown. We devise a set of numerical experiments that confirm that low frequencies are learned first, a behavior quantified by our definition.
Original languageEnglish (US)
Title of host publicationProceedings of the AAAI Conference on Artificial Intelligence
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Pages7168-7175
Number of pages8
ISBN (Print)1577358767
DOIs
StatePublished - Jun 28 2022
Externally publishedYes

Fingerprint

Dive into the research topics of 'A Computable Definition of the Spectral Bias'. Together they form a unique fingerprint.

Cite this