To improve the infrared small targets detection performance under complex background and noise interference, a single-stage infrared small target detection algorithm combining fully convolutional neural network and visual saliency is proposed. First, a lightweight fully convolutional neural network based on encoder-decoder architecture is designed to segment infrared images. The network can suppress the background and enhance targets simultaneously. Then, the saliency features of infrared small targets are used to further suppress false alarms. Finally, an adaptive threshold method is used to extract small targets. In the network structure, multiple subsampling layers are introduced to reduce computation load and increase the receptive field; multiscale features are introduced to improve the background suppression ability; attention mechanism is introduced to improve the training result of the model. Experiments on real infrared images show that the proposed algorithm is superior to the typical infrared small target detection algorithm with respect to detection rate, false alarm rate and computation time, and is suitable for infrared small target detection under complex background.