Gaussian mixture model in machine learning estimates the Gaussian distribution parameters of all constellation points according to the received signal. Therefore, a Gaussian mixture model cluster demodulation method is proposed for the nonlinear discrete Fourier transform spread orthogonal frequency division multiplexing system in visible light communications. The Gaussian distribution probability of each received signal constellation point is calculated, and the corresponding constellation point to the maximum probability is selected as the decision result of the received signal for demodulation so that signal-to-noise (SNR) ratio gain can be obtained. The simulation results show that the Gaussian mixture model cluster demodulation method can obtain 0.6 dB-2.7 dB and 0.2 dB-1.7 dB SNR gain for 16 and 32 quadrature amplitude modulation, respectively, in LED nonlinear channel.