A method for simulating the optical-coupler-based fiber compound ring cavity (CRC) filter for mode selection of the single longitudinal mode fiber laser is proposed. With this method, we theoretically simulated two types of novel dual-coupler double ring CRC (DCDR-CRC) filter and tri-coupler double ring CRC (TCDR-CRC) filter. By introducing the Vernier effect, the transmittance characteristics of the proposed two filters under different cavity length differences are analyzed. The effective free spectral range (FSR), suppression ratio (SR) and bandwidth of the main transmission peak of CRC filters are optimized by adjusting the coupling ratios, cavity length and cavity length difference of the DCDR-CRC and TCDR-CRC filters. The simulated results show that the optimized effective FSR of the ring cavity can effectively suppress the gain competition in the transmission passband of the wavelength selector. Furthermore the lower SR can suppress the gain competition between the adjacent transmission peaks of the CRC filter, and the narrower bandwidth of the main transmission peak can ensure that only one longitudinal mode of the laser can be selected.