In order to make full use of the frequency band resources in radio-over-fiber(RoF) system, a frequency multiplier tunable RoF system based on Mach-Zehnder modulator(MZM) is proposed, the techniques of wavelength selection, coherent detection, wavelength reuse and digital signal processing(DSP) are used to achieve the output of signals with different octave factors in the uplink and downlink of the RoF system. The simulation results show that when the RoF system transmits hexadecimal quadrature amplitude modulation(16QAM) signals, the multiple frequency multiplier factor can be realized in both the upstream and downstream links, and the minimum vector error ranges of downlink transmission and uplink transmission are 5.5% ～ 7% and 6.4% ～ 8.6% respectively.