Depth estimation from single fringe pattern is a fundamental task in the field of fringe projection three-dimensional (3D) measurement. Deep learning based on a convolutional neural network (CNN) has attracted more and more attention in fringe projection profilometry (FPP). However, most of the studies focus on complex network architecture to improve the accuracy of depth estimation with deeper and wider network architecture, which takes greater computational and lower speed. In this letter, we propose a simple method to combine wavelet transform and deep learning method for depth estimation from the single fringe pattern. Specially, the fringe pattern is decomposed into low-frequency and high-frequency details by the two-dimensional (2D) wavelet transform, which are used in the CNN network. Experiment results demonstrate that the wavelet-based deep learning method can reduce the computational complexity of the model by 4 times and improve the accuracy of depth estimation. The proposed wavelet-based deep learning models (UNet-Wavelet and hNet-Wavelet) are efficient for depth estimation of single fringe pattern, achieving better performance than the original UNet and hNet models in both qualitative and quantitative evaluation.