Pedestrian detection is a research hotspot and a difficult issue in the computer vision such as the Intelligent Surveillance System (ISS), the Intelligent Transport System (ITS), robotics, and automotive safety. However, the human body′s position, angle, and dress in a video scene are complicated and changeable, which have a great influence on the detection accuracy. Through the analysis on the pros and cons of Census Transform Histogram (CENTRIST), a novel feature was presented for human detection-Ternary CENTRIST (T-CENTRIST). The T-CENTRIST feature took the relationship between each pixel and its neighborhood pixels into account. Meanwhile, the relevancy among these neighborhood pixels was also considered. Therefore, the proposed feature description method can reflect the silhouette of pedestrian more adequately and accurately than that of CENTRIST. Secondly, a fast pedestrian detection framework based on T-CENTRIST in infrared image was proposed, which introduced the idea of extended blocks and the integral image. Finally, experimental results verified the effectiveness of the proposed pedestrian detection method.