Field outcrops are affected by natural conditions, and the outcrop surfaces are covered with vegetation and severely weathering, which makes the traditional lithology image recognition methods more challenging to implement. Combining artificial intelligence for rock image recognition lithology in the geological field has become an unavoidable trend with the advent of geological big data and the rising demand for intelligent geology. In this study, we propose SE-DeepLabv3+, an intelligent lithology recognition approach for multimodal clastic rock outcrop images based on an attention mechanism. The SE-DeepLabv3+ achieves more than 90% accuracy in lithology recognition when compared to classical classification methods and semantic segmentation methods, with hand annotation results as a reference, which is greater than other methods. For lithology identification, the SE-DeepLabv3+ was used on certain outcrop sections of the Qingshuihe-Karaza Formation along the southern boundary of the Junggar Basin in Xinjiang, and better identification results were obtained. The study employs UAV 3D image data, combined with artificial intelligence technology to identify the lithology of clastic outcrops, which can significantly enhance the efficiency of lithology identification, transform the conventional operation mode, and advance geological research toward quantification and intelligence.