Combining the advantages of OFDM technology with the coherent detection, the bit error rate(BER) performance of coherent OFDM FSO system was studied, and the influence of OFDM mapping mode and mapping order on the symbol error of the system was taken into account. In Gamma-Gamma atmospheric turbulence channel model, the bit error performance of three cases of weak, moderate and strong turbulence were taken into consideration separately from the perspective of the average signal-to-noise ratio and the required average received optical power. On this basis, the closed form expression of the symbol error rate was deduced. Simulation results show that the coherent detection OFDM FSO system can better overcome the effects of atmospheric turbulence, in weak and moderate turbulence, the system BER performance with QAM mapping mode is obviously better than that of PSK mapping mode, however, under the strong turbulence the advantage is not very obvious. In addition, with the increase of mapping order, system symbol error rate increases, so in practice it can improve the system BER performance by reducing the number of mapping order.