Aiming at the problem that the quality of the data stream transmission path in optical fiber network communication affected the utilization of network resources, an improved data transmission path optimization machine learning algorithm was proposed. Firstly, the machine learning was used to complete the preprocessing of the initial data, the data feature information was obtained, and the data stream classification was completed. Based on the analysis of the data flow within the optical fiber span, a cluster group was constructed to complete the adjustment of the data path and realize the full use of network resources. Secondly, the optimization of the cluster analysis was completed by taking the similarity matrix containing the characteristic parameters as the constraint condition. The similarity matrix was ​​established according to the data characteristic parameters, and the function mapping relationship was established between the characteristic parameters and the data flow type of the communication path. Finally, the kernel function was used to optimize the transmission path to realize the optimization of the network transmission path. The experiment optimized the path for a network containing multiple fiber spans, and compared it with the traditional K-means clustering algorithm. The ratio of the 6 different data streams in the test can fully reflect the data communication status under different conditions. The experimental results show that the classification accuracy of the algorithm is 94.6%, the average execution time is 12.8 s, and the average cluster change degree is 31.3%. The classification accuracy of the traditional K-means clustering algorithm is 84.6%, the average execution time is 20.8 s, and the average clustering change is 46.2%. The convergence time of this algorithm is also better than that of traditional algorithms, and it has higher accuracy and real-time performance in network data transmission.