To improve the precision of the encoder system, a method for error compensation in optical encoder was presented by applying Radial Basis Function (RBF). At first, how to integrate RBF model with error detecting system of encoder was introduced. The simulation analysis can be completed while run error detecting. At the same time, necessary model parameters could be obtained. Then, the parameters were written to DSP to run real error compensation. The optimization for RBF algorithms was introduced to resolve the problems existed in the RBF model, i.e., large amount of data, massive matrix operation and complicated arithmetic, ect. The practical application proves that the simulation effect is direct and clear by combing the model building with error detecting. The test results show that the optimized compensation model meets the requirements of running time without decreasing compensation precision obviously and the precision of system error is improved 1~3 times.