In the field of human following using mobile robot, human as the object for the detection is not the only requirement to be considered. Besides detection, continuous tracking and recognition is also another important criteria. In order to track and recognize a human, an effective algorithm needed to be implemented in the system. Speed Up Robust Feature (SURF) algorithm is a fast and robust algorithm for local, similarity invariant representation and comparison which mainly used for image analysis. In this paper we made the analysis using the SURF on the moving human target based on the shirts pattern and the results were shown in the discussion.