A method of adaptive threshold segmentation, detection and tracking for infrared sequence images is proposed to solve the problems that the current infrared target detection and tracking algorithm has weak scene adaptability, strong specificity and high false detection rate of small target in the first frame image under large field of view. The density, rectangularity and Hu invariant moments are selected as static variables to establish static decision criteria. The target moving speed, area and perimeter of target contour, adaptive segmentation threshold and location of ROI are selected as dynamic variables to establish dynamic decision criteria. The first frame target detection algorithm is used to calculate the target static variable and some of the dynamic features of the first frame image. The subsequent frame images are segmented by the improved local adaptive threshold segmentation algorithm and then the static and dynamic decision criteria are used to screen out the segmentation. Finally, the dynamic decision criteria are calculated and updated. The infrared target test results show that the method has good adaptability to different scenarios. By using this algorithm, the average tracking accuracy of the four scenarios is 95.81%, the average processing time per frame is 10.93 ms on microcomputer platform and 26.79 ms on embedded platform respectively.