Semantic segmentation, which classifies all pixels in an image and divides the image into several regions with specific semantic categories, is a key technology in the field of computer vision. In recent years, convolutional neural networks (CNNs) have been making breakthroughs and have demonstrated great potential in using deep learning to perform semantic segmentation. Herein, beginning with the definition of semantic segmentation, existing challenges in the field of semantic segmentation are discussed. Based on CNN principles, several datasets used for semantic segmentation algorithm evaluation are compared in detail, and recent deep learning methods based on decoders, information fusion, and recurrent neural networks in semantic segmentation are summarized. Finally, future development trends (e.g. enriching database scenes, improving real-time performance of algorithms, and researching the semantic segmentation) of three-dimensional point cloud data in semantic segmentation are summarized.