The maturity level and appearance defects of apples are crucial criteria for determining their quality. To automate the removal of immature and defective apples in picking tasks, a lightweight multi-task maturity classification model (L-MTCNN) is proposed. This model comprises two sub networks, D-Net and M-Net, for multi-task classification of apple appearance defects and maturity level. Furthermore, it uses a backbone network to extract feature information, which is then applied to D-Net and M-Net, thereby improving feature utilization and reducing overall recognition computation time. Introducing Triplet loss as the loss function for M-Net increases the separation between different maturity levels while reducing the variance within the same level. Additionally, based on industry standards, the study examines the appearance changes in various apple ripening processes and constructs an apple maturity dataset. A brightness-based color restoration algorithm is proposed to address the inconsistencies between collected apple images and their actual appearance, caused by varying lighting conditions during image acquisition. This algorithm restores the color restoration of the collected images and facilitates the creation of a reliable on apple maturity dataset. Experimental results indicate that D-Net and M-Net substantially improve average accuracy compared to AlexNet, ResNet18, ResNet34, and VGG16. Furthermore, in terms of recall rate, precision rate, and F1 score, the proposed model outperforms existing models in classifying maturity levels and defect statuses. This demonstrates that the model can achieve high-accuracy maturity level judgments for different types of apples, providing valuable insights for developing integrated operation robots.