This paper presents an image recognition method based on feature matching fusion and improved convolutional neural network. Aiming at the problem that the texture features extracted by the local binary pattern (LBP) descriptors are limited and cannot describe the image edge and direction information effectively, the feature extraction of the training set is performed in the convolutional neural network by the histogram of oriented gradient (HOG) and LBP hierarchical feature fusion method. Then the extracted feature pictures are input into the improved convolutional neural network for training and recognition. The simulations are performed on ORL, YALE and CAS-PEAL face databases with ReLU as the activation function and the output layer with the Softmax classifier, and trained on the TensorFlow framework. The recognition rate of the proposed method reaches 99.2%, 98.7%, and 97.2% respectively, which is higher than other algorithms for comparison.