To tackle the low classification accuracy caused by the diversity and distribution complexity of surface objects in small-sample datasets of remote sensing image scenes, this paper proposes a transfer learning based mixture of experts (TLMoE) classification model. The model can achieve more accurate scene classification by taking full advantage of the features from the convolution layer containing the local details and the fully-connected layer containing the global information of scenes through multi-channels. First, a pre-judgment channel based on the fully-connected layer features is established to preliminarily judge all kinds of scenes with global scene information; then exclusive expert networks are trained for each kind of scenes via the expert channel, which can mine the key local details contained in the convolution layer features of all categories of scenes targetedly and extract the local features used to distinguish the subtle differences between similar scenes to complete fine-grained identification. Finally, combined with the pre-judged weight, the model realizes the scene classification considering the global and local differences. Experiments on small-sample datasets show that the proposed method can effectively identify confusing scenes and achieve good classification results.