The Signal-to-Noise Ratio(SNR)of coherent detection system, which is affected by the non-linear effect of the photodetector, can not achieve the SNR level under the shot noise limited operation. Hence, there is an optimal local optical power to maximize the SNR. Aiming at the problem that the optimal local optical power of the system is difficult to determine, the theory of optimal local optical power based on balance detector is analyzed firstly. A method for determining the optimal local optical power which measure the heterodyne signal directly and calculate the SNR based on cross-correlation method and spectrum method is proposed subsequently. The experiments are conducted by setting up the experimental system for testing three type photodetector. The results demonstrate that the curve of measured SNR versus the local oscillator optical power basically consistent with the theoretical value, the optimal local optical power can be accurately measured for different detectors. The proposed method in this paper can be used to determine the optimal local optical power of a coherent detection system and improve the system SNR.