In view of the complexity of theoretical calculation and long calculation time of traditional evaluation method of optical fiber link transmission performance in coherent optical communication system, this paper uses machine learning technology to build correlation model and accurately evaluate the transmission performance of coherent optical transmission system with different modulation format and input optical power parameters. Through the training model, the results show that the machine learning model can replace the complex theoretical calculation, and can more accurately predict the predicted value, The overall average error between the theoretical value and the predicted Q value is 0.27 dB. The overall average error between the theoretical value and the predicted Optical Signal to Noise Ratio (OSNR) value is 0.33 dB. The calculation time of machine learning model is about 4.4 s, while the theoretical calculation time increases with the increase of the number of interference channels.