To enforce the visual effect and vision quality of infrared polarization images, the infrared polarization image enhancement algorithm based on mutual structure regularization is proposed. Relying on the description of infrared polarization features, the spatial local weighted gradient fusion strategy of Q component and U component in Stokes parameters is presented, and the polarized feature image is obtained, describing the boundary and contour information. Then, the mutual structure regularization constraint scheme is put forward. The gradient magnitude similarity map is applied to jointly regularize the boundary structure similarity between enhanced result and polarized feature image, meanwhile regularize the radiation consistency between enhanced result and radiant intensity image. Finally, the enhanced infrared polarization image with high quality is optimized. Experiments demonstrate that our mutual structure regularization algorithm can boost the visual contrast, visibility, and the polarization saliency of artificial targets in complicated background, with high engineering computational reliability.