This paper addresses the challenges of high model complexity and low classification accuracy in remote sensing image classification using convolutional neural networks. To overcome these challenges, a modified DeeplabV3+ network is proposed, which replaces the deep feature extractor in the encoder with lightweight networks MobilenetV2 and Xception_65. The decoder structure is also modified to feature fusion layer by layer in order to refine the up-sampling process in the decoding region. In addition, a channel attention module is introduced to strengthen the information association between codecs, and multiscale supervision is used to adapt the receptive field. Four networks with different encoding and decoding structures are constructed and verified on the CCF dataset. The experimental results show that the MS-XDeeplabV3+ network, which uses Xception_65 in the encoder and layer by layer connection, channel attention module, and multiscale supervision in the decoder, has reduced number of model parameters, faster training speed, refined edge information for ground objects, and improved classification accuracy for grassland and linear ground objects such as roads and water bodies. The overall pixel accuracy and Kappa coefficient of the MS-XDeeplabV3+ network reach 0.9122 and 0.8646, respectively, which show the best performance among all networks in remote sensing image classification.