A new algorithm was proposed to improve vibration-related fringes contrast for micro-vibration online detection based on time-average digital holography. First, phase change arising from the longitudinal micro-displacement of the object during vibrating process was corrected. Then phase noise caused by speckle noise was eliminated, and thus fringes pattern contrast that modulated by Bessel function was improved effectively. Finally, the amplitude distribution of the measuring surface was estimated according to Bessel function and fringe progressions. An experimental system based on time-average digital holographic interferometry was proposed for micro-vibration measurement that acoustic wave was used for excitation source. The experiment was verified by the reed that excited at the sinusoidal frequency of 0, 1 100, 2 000 and 9 000 Hz, respectively. The result based on vibration frequency of 2 000 Hz shows that the proposed method can improve vibration-related fringes contrast effectively. Amplitude of three-order zero points are 109.4 nm, 244.2 nm and 356.4 nm, and errors are 6.31%, 3.34% and 3.78%, respectively, compared with theoretical values, which demonstrate the effectiveness of the method within the allowable range.