TY - GEN

T1 - A Computable Definition of the Spectral Bias

AU - Kiessling, Jonas

AU - Thor, Filip

N1 - KAUST Repository Item: Exported on 2023-03-02
Acknowledged KAUST grant number(s): OSR-2019-CRG8-4033.2
Acknowledgements: This work was partially supported by the KAUST Office of Sponsored Research (OSR) under Award numbers OSR-2019-CRG8-4033.2.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.

PY - 2022/6/28

Y1 - 2022/6/28

N2 - Neural networks have a bias towards low frequency functions. This spectral bias has been the subject of several previous studies, both empirical and theoretical. Here we present a computable definition of the spectral bias based on a decomposition of the reconstruction error into a low and a high frequency component. The distinction between low and high frequencies is made in a way that allows for easy interpretation of the spectral bias. Furthermore, we present two methods for estimating the spectral bias. Method 1 relies on the use of the discrete Fourier transform to explicitly estimate the Fourier spectrum of the prediction residual, and Method 2 uses convolution to extract the low frequency components, where the convolution integral is estimated by Monte Carlo methods. The spectral bias depends on the distribution of the data, which is approximated with kernel density estimation when unknown. We devise a set of numerical experiments that confirm that low frequencies are learned first, a behavior quantified by our definition.

AB - Neural networks have a bias towards low frequency functions. This spectral bias has been the subject of several previous studies, both empirical and theoretical. Here we present a computable definition of the spectral bias based on a decomposition of the reconstruction error into a low and a high frequency component. The distinction between low and high frequencies is made in a way that allows for easy interpretation of the spectral bias. Furthermore, we present two methods for estimating the spectral bias. Method 1 relies on the use of the discrete Fourier transform to explicitly estimate the Fourier spectrum of the prediction residual, and Method 2 uses convolution to extract the low frequency components, where the convolution integral is estimated by Monte Carlo methods. The spectral bias depends on the distribution of the data, which is approximated with kernel density estimation when unknown. We devise a set of numerical experiments that confirm that low frequencies are learned first, a behavior quantified by our definition.

UR - http://hdl.handle.net/10754/689878

UR - https://ojs.aaai.org/index.php/AAAI/article/view/20677

UR - http://www.scopus.com/inward/record.url?scp=85146962469&partnerID=8YFLogxK

U2 - 10.1609/aaai.v36i7.20677

DO - 10.1609/aaai.v36i7.20677

M3 - Conference contribution

SN - 1577358767

SP - 7168

EP - 7175

BT - Proceedings of the AAAI Conference on Artificial Intelligence

PB - Association for the Advancement of Artificial Intelligence (AAAI)

ER -