Aiming at the problems of insufficient texture details, low contrast and loss of target information in the infrared and visible fusion image, an image fusion algorithm based on feature optimization and a generative adversarial network is proposed. Firstly, considering the impact of original image quality on the fusion results, an adaptive feature optimization module, which is guided by an objective function and based on the chameleon swarm algorithm, is designed, which can enhance the texture details of visible image and the contrast of infrared image. Then, in order to preserve more multi-modal information in the fused image, the generative adversarial network is introduced into the fusion framework. In the generator model, considering the difference in imaging mechanisms between infrared and visible images, a dual branch feature extraction network is constructed. At the same time, in order to solve the problem of insufficient feature extraction caused by single scale convolutional layer, a multi-scale dense connection module is designed to increase the range of network receptive field feature extraction, so as to comprehensively extract the deep semantic features and shallow texture features of images. To reduce the loss of important target feature information during the fusion process, a parallel attention model based on space and channel is designed in the feature fusion layer. By feeding the infrared and visible image feature information into the spatial and channel attention models, the correlation and dependency between different modal features are captured, and the expression ability of the network is improved, so as to better focus the thermal salient targets of infrared images and the texture details of visible images. In the discriminator model, to preserve the feature information of both infrared and visible images in the fusion results, a dual discriminator network structure for infrared and visible images is constructed, which can enable the fused image to retain as much rich information as possible from the original image in adversarial learning. Moreover, in the network training stage, in order to train a good model and enhance its robustness, 32 sets of images selected from the TNO image fusion data set are cropped and expanded to obtain 24 200 sets of infrared and visible image pairs. Finally, to verify the advantages of the proposed algorithm, subjective and objective results are compared with six classic fusion algorithms, such as DenseFuse, FusionGAN, ResNet-ZCA, MDLatLRR, PMGI and RFN-Nest. The experimental results show that the proposed algorithm has significant advantages in both subjective and objective evaluations. The fused images have richer texture details, clearer edges and targets, and better contrast. In the objective evaluation indexes, the entropy, spatial frequency, joint entropy, visual information fidelity, and gradient-based fusion performance obtain the optimal values, which are improved by 16.11%, 65.46%, 7.96%, 42.67%, and 33.24%, respectively, compared with the DenseFuse algorithm. In addition, to verify the effectiveness of the feature optimization module, multi-scale dense connection module and attention fusion module, ablation experiments are conducted on 21 sets of images and six evaluation indexes, all of which use the same parameter settings. The results of the ablation experiment indicate that compared with the original fusion network (i.e. none of the three modules is added), the six objective evaluation indexes are increased after the introduction of the three designed modules, which verifies the effectiveness of each module of the proposed algorithm.