The behavior recognition algorithm based on block matrix is proposed, which mainly includes six kinds of abnormal behavior of jumping, accelerated running, falling down, squatting, waving and bag. Firstly, inter frame difference method is applied to on video stream to extract contour features of the moving object and the mathematical morphology method was carried out to deal with it. Then block bilateral two dimensional linear discriminate analyses are applied to extract features for the obtained contour. Finally, nearest neighbor classifier is used for classification through template matching method. Experiment results show that the method has a certain practical value in the identification of the abnormal behavior given samples in this paper.