In order to overcome the tracking difficulty caused by distorted shape and motion law of the target of non-similar imaging in infrared wide-field staring system, a gradient entropy weighted centroid model is proposed. The multi-targets tracking is achieved by combing Kalman filters with this model. The model adopts gray value gradient to describe the probability distribution of dim target. The gradient entropy weights in 8 directions of target are obtained to indicate the correlation between the different regions and centroid of the target. Meanwhile, multi-direction maximum gradient method is proposed to extract the sky-ground line in wide field and obtain the position of the deep space background. Then, the starting point is detected through background suppression. The linear path planning is adopted to solve the mutual interference of Kalman filtering models when the targets meet. The experiment results show that the proposed method can track aeromodellings, airplanes and missiles in three environments in field view of 70° and 180°. Compared with the traditional tracking method, the proposed method has a lower center location error and a high accuracy, and the average frame rate is increased by about 1.5 times.