The terrain segmentation in polarimetric synthetic aperture radar (PolSAR) images is an important task for image interpretation. Since the speckle noise and complex scattering mechanism exist in SAR images, the classification results achieved by traditional methods appear fragmented. Gradually, deep-learning-based methods are proposed to solve this problem. However, only the amplitude data in the SAR image is utilized, which limits the classification precision. In this letter, a novel method based on a dual-attention fusion network (DAFN) is presented. DAFN is mainly composed of a two-way structure encoder for feature extraction and the attention-based fusion module. Considering the terrain characteristic and the SAR imaging mechanism, the introduction of the polarization information in DAFN increases the discrimination of different categories, which contributes to the consistent and accurate fine-grained classification results. To demonstrate the effectiveness of the proposed method, the corresponding experiments are done based on a GaoFen-3 satellite full-polarization SAR data set, in which the superior performance in terrain segmentation is obtained.