A face recognition algorithm based on fusing Gabor feature and local ternary derivative pattern (LTDP) is proposed. The LTDP method is improved and an adaptive threshold operator is presented in this algorithm. Face images are filtered by the Gabor filter to extract face magnitude features with multi-scale and multi-orientation. The local Gabor adaptive ternary derivative pattern (LGATDP2) method is used to extract the local orientation relationship patterns of magnitude feature images. The information entropy weight is calculated by the pattern area histogram statistics, and the face image is described after the weighted series. The similarity match of the feature histograms is carried out with the chi-square distance. The simulations are conducted in the ORL and the Yale databases. The results show that the LGATDP2 method has better universality, and it is more robust under the conditions of illumination variation, expression changing and noise interference.