In the modulation profilometry, the fringe projection axis coincides with the observation axis, and the height information of measured object is encoded into fringe defocus, which completes the measurement with complex surface, the problems of shadows, shutoff and phase truncating. However, in the practical measurement, extraction of modulation distribution from fringe patterns will be affected on account of high-order harmonics produced by photoelectric response nonlinearity of CCD detector, which will reduce the measurement accuracy. The wavelet transform method is applied to modulation retrieval, and the frequency-domain description analysis is deduced for the fringe pattern influenced by nonlinearity. With the advantages of local analysis and multi-resolution, an adaptive optimized filtering operation is adopted to demodulate the image, which mitigates the effect of nonlinearity on measurement accuracy effectively and obtains better reconstruction results compared with Fourier transform method, both the computer simulation and practical experiment verified the validity of the proposed method.