In order to solve the problem that the traditional frequency domain least mean square (FD-LMS) algorithm will lose efficacy with the increase of differential mode group delay (DMGD) when the algorithm is used for demultiplexing of the 6×6 mode division multiplexing (MDM) system, an improved FD-LMS demultiplexing algorithm is proposed. By improving the error signal calculation method, the convergence performance of the output signal of the equalization filter is improved, and the steady-state error of the algorithm is reduced. Besides, the equalization performance of the traditional FD-LMS algorithm is compared with the improved FD-LMS algorithm. Simulation results show that the improved FD-LMS algorithm has great advantage over the traditional FD-LMS algorithm in demultiplexing performance on the premise that the computation complexity does not significantly increase. The optical signal to noise ratio (OSNR) penalty of the improved FD-LMS algorithm is 2.6 dB lower than that of traditional FD-LMS algorithm at a transmission distance of 80 km with DMGD is 50 ps/km.