In order to study the interference stability in phase coded quantum key distribution (QKD) system, the influence that optical fiber birefringence, temperature variation, asymmetry and coupling ratio have on the interference visibility of the system is studied in details in this paper by using analytic and numerical methods. Further more, the influence of temperature variation and that of asymmetry could be eliminated and depressed respectively by adjusting of phase modulator in receiver. But the influence of birefringence could not be amended. In addition, the influence of the coupling ratio of coupler is limited to system efficiency only.