An infrared small target detection algorithm based on adaptive filter is proposed, which consists of twolayer Two Dimensional Least Mean Square Filter (TDLMS) filters: background removal and target extraction. The background template and target template are introduced, and the original infrared image is processed by TDLMS filtering method. The background noise and clutter are eliminated effectively, while the target information is preserved as much as possible. In the implementation of TDLMS algorithm, an automatic stepsize adjustment algorithm is proposed. The optimal weights of TDLMS filters can be iterated by adjusting the adaptive stepsize through the statistical parameters of the image, and then fused with the weight template, which effectively enhances the detection effect of the target. The experimental results show that the method can adapt to target detection under different background, and effectively improve the detection performance of small infrared targets.