The choice of linear polarization mode in optical fiber is a key factor affecting transmission quality of mode division multiplexing system. Firstly, a direct detection optical fiber transmission system based on hybrid multiplexing of mode division and wavelength division is designed. Secondly, based on the theory of linear polarization mode in fiber, the transmission characteristics of LP01, LP11, LP02, LP12 and LP21 five modes in the optical fiber transmission system are simulated by Optisystem15.0 software. Then, four modes of LP01, LP11, LP02 and LP12 are multiplexed, and the simulation results show that when the core radius of graded-index multimode fiber changes in the range of 19~25 μm, the optimal core radius for each channel with Q factor above seven is 21~22 μm, and the optimal value does not change with the transmission distance. Finally, in the eight-channel hybrid multiplexing system, the direct detection method is used to realize the short distance transmission with bit error rate below 10-9 at 2.0 km.