Aiming at the problem that the attitude measurement accuracy of Fine Guidance Sensor (FGS) was affected by the error of star point extraction system, a high-precision star point positioning system error compensation method based on Gradient Boosting Decision Tree(GBDT) fitting method was proposed. In order to solve the problems of less fitting samples and large differences in input characteristics, a decision tree that was insensitive to the input range and easy to train was used as the base model. Combining the boosting method in ensemble learning to generate a new base model to obtain the functional relationship between the systematic error and the detector fill rate, sampling window size, Gaussian width of star image and star point centroid coordinate calculation value, and based on this function relationship to the star point centroid. The coordinate estimate was systematically corrected. The experimental results show that compared with the support vector regression machine, the error of the high-precision star point localization algorithm based on GBDT is reduced by 60.6%. The corrected centroid error is 0.014 5 pixel, and the error is reduced by 61.5%.