The effects of mode coupling and difference mode group delay on signal transmission in mode-division multiplexing system can be compensated with equalization to realize signal reliable transmission. Compared with time-domain equalization, frequency-domain equalization can effectively reduce digital signal processor (DSP) complexity. FD-CMA (frequency-domain constant modulus algorithm) and FD-MMA (frequency-domain multi-modulus algorithm) are used to simulate and compare in 2×56 Gb/s 16QAM mode-division multiplexing system. The results show that the equalization performance of FD-MMA is better than FD-CMA in constellation diagram and bit error rate (BER) performance. At the error bit rate of 10-3, FD-MMA has a 2.8 dB improvement of optical signal-to-noise ratio than FD-CMA.