The study on attention in attention network (A²N) in single-image super-resolution has revealed that all attention modules are not beneficial to the network. Therefore, in the design of the network, input features can be divided into attention and nonattention branches. The weights on these branches can be adaptively adjusted using dynamic attention modules based on the input features so that the network can strengthen useful features and suppress unimportant features. In practical applications, lightweight networks are suitable to be run on resource-constrained devices. Based on A²N, the number of attention in attention block (A²B) in the original network is reduced and lightweight receptive field modules are introduced to enhance the overall performance of the network. In addition, by adjusting the L1 loss to a combination loss based on Fourier transform, the spatial domain of the image is transformed into the frequency domain, enabling the network to learn the frequency characteristics of the image. The experimental results show that the improved A²N reduces parameter count by about 25%, computational complexity by about 20%, and inference speed by 15%, thereby enhancing the performance.