Instrument transfer function(ITF) of white light interferometer(WLI) influences the measurement accuracy of power spectral density(PSD) for optical surfaces. To understand the nature of ITF, WLI was treated as an incoherent optical system and a sinusoidal surface was the input signal. Studying the Bessel function expansion of interference intensity spectrum of sinusoidal surface, the theoretical analysis indicated that the low pass of optical system led to the surface height attenuation. The nonlinearity of Bessel function resulted in the nonlinear response of WLI to height, and the response approached linear when the height was much less than wavelength. Numerical simulations and experiments validated the theoretical conclusions. Step-artefacts of 30 nm, 80 nm and 120 nm height were employed to calibrate the ITF of a commercial WLI. The experimental result and the simulation result match each other well. ITF rises when the surface height increases, which manifests obvious nonlinearity. If the surface height is less than λ/10, ITF nearly equals the modulation transfer function of WLI optical system, in which case WLI is approximately linear to surface height. This paper has promotion for understanding WLI transfer characteristic and enhancing PSD measurement accuracy.