The kernel ridge regression classifier used in adaptive color attribute tracker makes little use of information about the context, which may easily result in drifting in cases of fast target motion, partial occlusion or background clutter. To solve the problem, the interference peak position is determined at first by calculating the peak side lobe ratio of the response matrix. Then, the corresponding context constraints are introduced on the kernel ridge regression classifier to enhance the background utilization. Finally, a scale space filter is established for supporting multi-scale variation. The experimental results based on the Visual Tracker Benchmark dataset show that this method can achieve high tracking accuracy with a mild loss of frame rates.