Deep Reinforcement Learning Based Beamforming Codebook Design for RIS-aided mmWave Systems

Reconfigurable intelligent surfaces (RISs) are envisioned to play a pivotal role in future wireless systems with the capability of enhancing propagation environments by intelligently reflecting the signals toward the target receivers. However, the optimal tuning of the phase shifters at the RIS is a challenging task due to the passive nature of reflective elements and the high complexity of acquiring channel state information (CSI). Conventionally, wireless systems rely on pre-defined reflection beamforming codebooks for both initial access and data transmission. However, these existing pre-defined codebooks are commonly not adaptive to the environments. Moreover, identifying the best beam is typically performed using an exhaustive search that leads to high beam training overhead. To address these issues, this paper develops a multi-agent deep reinforcement learning framework that learns how to jointly optimize the active beamforming from the BS and the RIS-reflection beam codebook relying only on the received power measurements. To accelerate learning convergence and reduce the search space, the proposed model divides the RIS into multiple partitions and associates beam patterns to the surrounding environments with low computational complexity. Simulation results show that the proposed learning framework can learn optimized active BS beamforming and RIS reflection codebook. For instance, the proposed MA-DRL approach with only 6 beams outperforms a 256-beam discrete Fourier transform (DFT) codebook with a 97% beam training overhead reduction.