The image semantic description model usually adopts the encoder-decoder method to realize the image semantic description. The model has problems such as insufficient utilization of image features and insufficient location information extraction of image objects. In response to this problem, an image semantic description algorithm is proposed that integrates the attention mechanism in the encoder part, and the attention weight of different image features is allocated through the context information of the decoder, thereby improving the expressive ability of image semantic description. And verified on the Flickr30k and MSCOCO data sets, the model improves the BLEU-4 evaluation index by 1.9% and 0.8%, respectively. The experiment proves the effectiveness of the proposed algorithm.