Although guided image filtering (GIF) is known for preserving edges and fast computation, it may produce inaccurate outputs in depth map restoration. In this paper, a novel confidence-weighted GIF called mutual-structure weighted GIF (MSWGIF) is proposed, which replaces the mean filtering strategy in GIF during handling overlapping windows. The confidence value is composed of a depth term and a mutual-structure term, where the depth term is utilized to protect the edges of the output, and the mutual-structure term helps to select accurate windows during the structure characteristics of the guidance image are transferred to the output. Experimental results show that MSWGIF reduces the root mean square error (RMSE) by an average of 12.37%, and the average growth rate of correlation (CORR) is 0.07% on average. Additionally, the average growth rate of structure similarity index measure (SSIM) is 0.34%.