In this paper, based on the leg movement mode analysis, a novel human computer interaction (HCI) scheme with which users can interact with computer is presented. Firstly, an efficient valley point location algorithm is proposed to detect and separate the left and right leg, then an extended Mean Shift tracking algorithm with Kalman filter is applied to track the leg's movement, finally the movements are grouped into six classes to control the game. The application in the game controlling proves that the presented scheme is robust in leg movement mode classification and real-time in game controlling.