The optical spectrum details could be detected theoretically even as low as MHz lever based on heterodyne interference principle. A scheme design of optical spectrum detecting system based on heterodyne interference technique was introduced, in which the signal transmission principle from the 90°optical mixer to the balanced detector was deduced. Moreover, the relationship between the settings of the local oscillator, as well as the filter intermediate frequency and the system resolution was concluded. The compact spectrum detecting system is built upon OptiSystem software, in which the wavelength sweep and iteration could be used to simulate actual local oscillator operating mode, thus the feasibility and superiority of the ultra-high resolution spectrum analysis function were verified. In the end, the simulation results of different linewidth and scanning step length of local oscillator, as well as the filtering parameters was given in 40 MHz frequency interval. The resolution requirement on the local oscillator and filter was concluded.