In this study, a new method is proposed by improving an existing deep-learning network, where aerial high-resolution hyperspectral data and LiDAR data are combined for the fine classification of tree species. First, feature extraction and fusion are performed for different data sources. Subsequently, a classification network named CA-U-Net is constructed based on the U-Net network by adding a channel-attention-mechanism module to adjust the weights of different features adaptively. Finally, we attempt to address the problem of low identification precision for small-sample species by modifying CA-U-Net in class-imbalance cases. The research results show that 1) the CA-U-Net network performs well, with an overall classification accuracy of 96.80%. Compared with the FCN, SegNet, and U-Net networks, the CA-U-Net network shows improvements of 8.56, 11.99, and 3.31 percent points, respectively, in terms of classification accuracy. Additionally, the network exhibits a higher convergence speed. 2) Replacing the original loss function in the CA-U-Net network with a cross-entropy loss function based on the class-sample-size balance can improve the classification accuracy for tree species with fewer samples. The proposed methodology can serve as an important reference in small-scale forestry, such as orchard management, urban-forest surveys, and forest-diversity surveys.