As the measuring accuracy of a light open 3D vision coordinate measuring system depends on the accurate center position of a probe tip, this paper proposed a novel and simple approach based on the model of measuring system for calibrating the center position of probe tip for the light pen. Based on position invariant principle, the objective function was established according to the control point information of every image. Then, the general inverse method for least square solution of nonlinear multivariable equations was employed to solve the object function and to obtain the center position of probe tip. Finally, average theory was used to improve the stability and convergence speed and to implement the calibration with a reference standard cone. The experimental results demonstrate the effectiveness of the proposed approach. Under a satisfied convergence speed, the repeatability of the system reaches 0.033, 0.030 and 0.043 mm in x, y and z axes, respectively. A comparison on the measured circle diameter of a workpiece and a reference shows that the system measurement accuracy has met medium precision industrial measurement requirements with the calibrated probe tip center.