Recently, increasing interest in managing pedestrian and bicycle flows has been demonstrated by cities and transportation professionals aiming to reach community goals related to health, safety, and the environment. Precise forecasting of pedestrian and bicycle traffic flow is crucial for identifying the potential use of bicycle and pedestrian infrastructure and improving bicyclists’ safety and comfort. Advances in sensory technology enable collecting massive traffic flow data, including road traffic, bicycle, and pedestrian traffic flow. This paper introduces a novel deep hybrid learning model with a fully guided-attention mechanism to improve bicycles and pedestrians’ traffic flow forecasting. Notably, the proposed approach extends the modeling capability of the Variational Autoencoder (VAE) by merging a long short-term memory (LSTM) model with the VAE’s decoder and using a self-attention mechanism at multi-stage of the VAE model (i.e., decoder and before data resampling). Specifically, LSTM improves the VAE decoder’s capacity in learning temporal dependencies, and the guided-attention units enable selecting relevant features based on the self-attention mechanism. This proposed deep hybrid learning model with a multi-stage guided-attention mechanism is called GAHD-VAE. Proposed methods were validated with traffic measurements from six publicly available pedestrian and bicycle traffic flow datasets. The proposed method provides promising forecasting results but requires no assumptions that the data are drawn from a given distribution. Results revealed that the GAHD-VAE methodology can efficiently enhance the traffic forecasting accuracy and achieved better performance than the deep learning methods VAE, LSTM, gated recurrent units (GRUs), bidirectional LSTM, bidirectional GRU, convolutional neural network (CNN), and convolutional LSTM (ConvLSTM), and four shallow methods, linear regression, lasso regression, ridge regression, and support vector regression.