In order to improve the transmission capacity of the optical communication system, this paper proposes a new type of 12 channels wavelength-mode-polarization division hybrid multiplexing system, and the wavelength division multiplexing device is combined with the mode division multiplexer, so that the system can be adapted to the wavelength-mode division hybrid multiplexing. Signals of 12 channels are transmitted by the few-mode fiber, and the output signal is decompressed with a constant mode algorithm. Through the VPI transmission Maker simulation platform, the influence of wavelength interval, transmitter output power and laser line width on the performance of the system is analyzed. The simulation results show that under the condition that the signal-to-noise ratio is 20 dB and the transmitter output power is 0 dBm, using 2 wavelengths and 6 modes, each mode carries the 25 Gb/s data, and the bit error rate is in the 10-4 volume level to realize the 25×2×6 Gb/s polarization division multiplexing-binary quadrature amplitude modulation(PDM-4QAM) signal transmits 10 km without relaying.