Aiming at the influence of matched filter (e.g. root raised cosine filter) on the coherent optical fiber communication system, the effect of roll-off factor (Rf) are studied under different value of optical bandwidth, residual dispersion and nonlinearity. Simulations are carried out for 256 Gbit/s Polarization Division Multiplexing-16 Quadrature Amplitude Modulation (PDM-16QAM) signals when the optical bandwidth are 33, 50 and 100 GHz, the residual dispersion ranging from -300 to 300 ps/nm and launched power ranging from -3 to 4 dBm in a 1 200 km optical fiber link. The simulations show that a good transmission performance can be achieved for limited optical bandwidth system when Rf is set to the value of (1+Rf) B (B is the Baud rate of the signal) that similar to the system optical bandwidth. When optical bandwidth near or exceed 2 times Baud rate of signals, Rf should be set between 0.2 and 0.5 to achieve a better transmission performance.