Low Power Hardware Architecture for Sampling-free Bayesian Neural Networks inference

Standard NNs should not be employed mindlessly in critical applications due to their incapability to express the uncertainty of their predictions. On the other hand, Bayesian Neural Networks (BNNs) can measure the uncertainty of their predictions. There are two methods for BNN inference, the Monte Carlo-based method, which requires the sampling of weights distributions and multiple inference iterations, and moment propagation, where the mean and variance of a normal distribution are propagated through the BNN. Hardware implementations of moment propagation BNN inference consume less power than Monte Carlo because they complete the inference in a single forward pass. However, because the propagation of distribution moments through nonlinear activation functions leads to large hardware designs, these functions are usually approximated by polynomials. Hardware implementations of moment propagation have been studied solely for fully-connected neural networks while lacking optimal accuracy due to the approximation of the ReLU activation function with a single polynomial term. Therefore, in this work, we add one more polynomial term in the approximation of ReLU, providing better accuracy with negligible additional hardware. We also propose a polynomial approximation for another common activation function, tanh, and extend the hardware implementation to Convolutional Neural Networks (CNNs). Experimental results demonstrated that the proposed approximation of ReLU outperforms the previously suggested single-term polynomial by achieving up to 5.9% higher accuracy with merely up to 0.029 W power overhead.