In order to solve the high cost problem caused by high peak to average power ratio(PAPR) which need the high-resolution digital-to-analog converter/analog-to-digital converter(DAC/ADC) in the intensity modulation and directly detection (IMDD)-based universal multi-carrier(IMDD-UFMC) modulation optical transmission system, a non-uniform quantization scheme based on self-organizing mapping(SOM) neural network is proposed in this paper. The scheme minimizes the quantization error by unsupervised learning through a competitive mechanism to realize the non-uniform quantization with the minimum quantization noise, so as to improve the system performance of IMDD-UFMC. In this way, the low-bit resolution but efficiency DAC can be employed in the IMDD-UFMC system. The simulation results show that compared to the uniform quantization scheme, the non-uniform quantization scheme of SOM can reduce 1 bit resolution in the case of the same system performance, in some cases, it can improve the receiving sensitivity of at least 1.7dB.