To improve the classification performance of hyperspectral images with unbalanced, few-labeled samples, an improved semantic autoencoder network is proposed in this paper. This network first introduces hyperspectral category-label information into the semantic autoencoder model, establishing the association between known and unknown categories by mapping the original data and label information of different datasets to the same feature space. It then maps the training dataset features to the unified embedding space to learn the correspondence between the visual features and the semantic features of the category labels. Finally, an objective function based on a graph regularization term is constructed to preserve the feature manifold structure in the dataset, and the global problem is decomposed into several smaller, more manageable local subproblems using the alternating direction multiplier method to obtain the global optimal solution. Three hyperspectral datasets with different spectral dimensions, numbers of spectral bands, and land cover types were selected to ensure the diversity of the experimental data. The results showed that the proposed method achieved better classification accuracy with a small number of labeled samples compared with other state-of-the-art methods, making it suitable for the engineering classification of unbalanced hyperspectral image data.