According to the features of infrared point target in the sky, a detection algorithm is presented. Firstly, contrast of the images are enhanced by high-pass filter, and then the coarse detection is performed based on differentiating two neighboring frames in image sequence to extract candidate target regions. Secondly, the original image or the enhanced image is segmented in the candidate regions, so the possible point targets are detected. Finally, the true target can be accurately detected by removing noise interference based on continuity of moving target in the image sequence. The effectiveness of the algorithm is proved by the experimental results for several real infrared image sequences.