This paper presents a framework for classification and recognition of human activities in complex motion. We propose a template matching based method to classify the objects and a rule-based approach to recognize human activities. First, moving objects are detected and their silhouettes are generated in each frame. Second, template matching based approach is used to classify the generated silhouette and then a rule based classifier is applied to classify human activities such as running, walking, bending, boxing and jogging etc. The experimental results show that the system can recognize seven types of primitive actions with high accuracy.