This paper presents a self-adapting algorithm based on Mean Shift model to track the target in video sequences. Firstly, two-dimensional histogram is used to represent the target instead of one-dimensional histogram, so as to better distinguish the target from background. Secondly, algorithm has been improved by adding self-adapting progress to remove errors caused by local maximum. Experiments on several video sequences showed that the proposed algorithm performs of high accuracy and good robustness to handle target tracking where background objects are similar to target, and can be applied on a real-time system.