Moving object tracking is one of the key technologies in video surveillance. Mean shift algorithm fails to track the moving object in complicated environment. In this paper, a new strategy is proposed to improve the tracking ability of mean shift algorithm, in which the contrast between object and background along with similarity evaluation are applied for generating and updating object model. To eliminate the interference of the most similar features between tracking object and background, the coefficient ratio of the object to surrounding environment is first imported to generate the object model. To make sure the accuracy of updating object model, the effective way that combines similarity evaluation and Kalman filtering prediction is then applied for judge whether the tracking object is sheltered by other objects or background. The experimental results have shown that the proposed method can tack the moving object stably.