The Mean Shift procedure is a popular object tracking algorithm since it is fast, easy to implement and performs well in a range of conditions. However, classic Mean Shift tracking algorithm fixes the size and orientation of the tracking window, which limits the performance when the target’s orientation and scale change. In this paper, we present a new human tracking algorithm based on Mean Shift technique in order to estimate the position, scale and orientation changes of the target. This work combines moment features of the weight image with background information to design a robust tracking algorithm entitled Scale and Orientation-based Background Weighted Histogram (SOBWH). The experimental results show that the proposed approach SOBWH presents a good compromise between tracking precision and calculation time, also they validate its robustness, especially to large background variation, scale and orientation changes and similar background scenes.