The change in the facial features with age is a crucial factor affecting the performance of face recognition systems. Therefore, this paper proposes a cross-age face recognition method based on a Transformer. First, the improved T2T-ViT model was used to extract mixed features considering the age and identity. The extracted age and identity features were obtained through residual factor decomposition. Subsequently, the correlation between the age and identity features was removed using a decorrelated adversarial learning algorithm with linear feature decomposition to achieve age-invariant face recognition. Compared with the convolutional neural network-based DAL and MTLFace methods, the improved model significantly reduces the number of model parameters, multiply-add operations (MACs), and calculation time. Finally, the effectiveness of the proposed method is verified using the recognition results on benchmark datasets, AgeDB-30, CACD_VS, CALFW, and LFW, and the accuracy of the proposed method is comparable to that of the DAL and MTLFace methods for age-invariant face recognition.