Quantum-noise randomized cipher (QNRC) is an emerging physical layer encryption scheme to improve the security of optical fiber transmission. Based on the wire-tap channel model, the transmission performance and security performance of phase shift keying (PSK) QNRC system are evaluated quantitatively with legal channel capacity and information interception probability as the performance metric, and the effects of mesoscopic power, mechanism and number of quantum state masked by quantum noise on system performance are also analyzed. The simulation results show that in order to improve the transmission performance and security performance of the system, it is necessary to use optical amplifier at the transmitting end. While ensuring the transmission performance of the system, the mesoscopic power can be reduced appropriately to improve the system′s security. Higher mechanism can improve the security of PSK-QNRC system effectively without affecting its transmission performance. The security of data is mainly limited bythe number of quantum states masked by quantum noise.