Tracking is considered as a binary classification problem in this paper, and a novel contour tracking algorithm is proposed based on Adaboost ensemble learning and fast level set. First, an ensemble of weak classifiers is trained online to distinguish between the target and the background. Then, the ensemble of weak classifiers is combined into a strong classifier using AdaBoost and the strong classifier is used to label pixels in the next frame as either belonging to the object or the background, so the velocity function of fast level set is obtained. Contour tracking is realized by evolving the zero level set curve using dynamic neighbor region fast level set algorithm which is proposed in this paper. Temporal coherence is maintained by updating the ensemble with new weak classifiers that are trained online during tracking. Experiments show that this algorithm can track the target contour under the conditions of moving background, illumination variation, partial occlusion and the scale change of target.