The Kerr nonlinear effect limits the performance of the system. A nonlinear suppression scheme combining Probability Shaping (PS) and Phase Conjugate Twin Wave (PCTW) is proposed to improve the performance of large capacity optical fiber communication system. With co-simulation of Optisystem software and Matlab software, a PCTW-PS system is built in 16 Quadrature Amplitude Modulation (16QAM) and 64QAM modulation formats. The system consists of two kinds of PS (Huffman coding, sign inversion coding) and three dimensions PCTW (time domain, polarization domain, sub-carrier domain) to form different combinations. The simulation results show that the minimum Bit Error Rate (BER) of the scheme combining PCTW and PS can be greatly reduced. The BER of 16QAM time domain PCTW scheme is smaller than other schemes. Therefore, it can be seen that both PS and PCTW can improve system performance. The combination of PS and PCTW can further improve the performance. The 16QAM-PCTW scheme in time domain has better performance than other schemes. The simulation results also show that the BER of the in-phase component I and the quadrature component Q is different in PCTW system. Therefore, the concept of PCTW-IQ imbalance is proposed and the imbalance is especially obvious for polarization domain PCTW scheme.