The ability of active infrared thermography to detect delamination damage in different matrix composites has not been evaluated. Two typical thermosetting/thermoplastic composite laminates were designed and manufactured in this paper. The three methods of pulse infrared thermography, ultrasonic infrared thermography, and ultrasonic C-scan were used to detect and study the delamination damage under different impact energy. The scanning results of ultrasonic C-scan were for reference, and the detection results of the two infrared thermography were compared. At the same time, a region growth algorithm based on the similarity theory of image intensity value was developed for the threshold segmentation and extraction of damaged regions of heat map sequence. Accuracy and quantitative damage identification results show that pulse infrared thermography is effective in detecting layered damage of thermoset composites, but it is not suitable for damage detection of thermoplastic composites. Ultrasonic thermal thermography is effective for both types of delamination damage of composites. It has better detection capability and overall detection accuracy is better than pulse infrared thermography. The deep mechanism of different damage detection effects was analyzed, and the evaluation process and standard of infrared thermography for the two matrix types of composites were proposed.