To solve the problem of the phase recovery of a single electronic speckle interferometric fringe pattern, we propose a USS-Net, which combines a subpixel convolution module and a structured feature enhancement module to realize end-to-end phase recovery of a single fringe pattern using U-Net as the basic network. First, the upsampling method of U-Net is improved, and the subpixel convolution module is used to make the proposed network learn more fringe details while reducing the influence of deconvolution zero filling on gradient calculation. Second, in the coding part, the feature fusion method of U-Net is improved, and the structured feature enhancement module is used to fully integrate feature information with different scales. Hence, the proposed method can solve the problem of poor feature extraction caused by uneven fringe density and increase the segmentation accuracy for a single pixel point. The electronic speckle pattern interferometry (ESPI) fringe-phase simulation and experimental datasets are established, and the USS-Net model is tested and analyzed to verify the effectiveness of the proposed method. The proposed method overcomes the shortcomings of traditional phase recovery methods, such as cumbersome processes and high susceptibility to noise disturbance, and effectively increases the accuracy of phase recovery of a single fringe pattern.