Based on the analysis of scale space theory and current Mean Shift algorithm,a scale and diretion adaptive Mean Shift tracking algorithm is proposed.The method obtains the target areas according to the variance between target and background,and calculates the best ellipse description based on the scale space theory.The best ellipse parameters of the preview frame is used as the original value of the current frame,and the ellipse parameters can be updated according to the Mean Shift principle.The experimental results show that the new Mean Shift tracking algorithm can adapt to the changes of scale and direction of the object effectively.