Action recognition has always been an important task in the field of computer vision. There are mainly two tasks based on RGB video and human skeleton. The mainstream methods are 3D convolutional neural network and graph convolutional neural network. For the data modality of human skeleton, this work designs a graph convolutional neural network based on the self-attention mechanism. The algorithm can achieve advanced performance on skeleton-based action recognition tasks. In addition, a method is proposed to use deep supervision methods to supervise the intermediate features of video and human skeleton, which improves the coupling of the two data features and further improves network efficiency. The network structure of this algorithm is simple, and only 3.37×10⁷ parameters are used to achieve an accuracy of 95.6% on the NTU-RGBD60 (CS) dataset.