To address the issues of low efficiency and accuracy for flue cured tobacco area extraction in remote sensing images, we propose a novel tobacco area extraction model with remote sensing images. The proposed model integrates the lightweight ASPP and U-Net framework for improving performance. The model enhances its performance in three ways: 1) integrating a lightweight atrous spatial pyramid pooling module at the junction of the U-Net encoding and decoding layers, 2) substituting the ReLU activation function with ReLU6, which can compress the dynamic range during low-precision computation and enhance the algorithm's robustness, and 3) optimizing segmentation results by developing a post-processing algorithm that constructs labeled maps through morphological hole filling. To demonstrate the effectiveness of the model, we utilize UAV remote sensing images as the experimental dataset. We also conduct comparative experiments with traditional semantic segmentation models and ablation study of the proposed model. Compared to traditional semantic segmentation algorithms such as FCN, U-Net, SegNet and DeepLabV3+, the proposed method achieves better performance, where the pixel accuracy and the mean intersection over union of the proposed model are 93.7% and 84.1%, respectively. Furthermore, the model enhances computational speed while maintaining the accuracy of segmentation.