A novel infrared polarization target detection method based on matrix recovery and double window filter, was proposed to remove the complex desert background clutter in the detection of dim and small target. Data analysis demonstrated that similarity of infrared polarization Q image background was the highest. Using morphological filter enhance the similarity of background. Q image background could be seen as a low-rank matrix, dim and small target could be seen as redundancy data which break the similarity of background. So background suppression mathematics model based on stable matrix recovery was build. Detection was completed after double window filter segments targets in background suppression result. Experimental results on real-world infrared polarization images and comparisons with state-of-the-art methods can demonstrate the effectiveness and robustness of the proposed method, and it was suitable for engineering application.