Tracking non-rigid objects with significant shape variation in complex scenario is a difficult problem. Human tracking is a special case of this problem since human body has good local rigid properties. In this paper, we propose a novel human tracking method which explores the local rigid properties while keeping the global structure very well. This method consists of three stages. First, the human body is represented by structured rigid parts extracted using patches clustering method. Then, the rigid parts are tracked through a structured constraint method. Finally, the optimal estimated state of the object is obtained through global similarity. Experimental results show that the proposed method has good performance for human tracking with big posture and shape change.