Traditional multi-class classifying methods treat outputs separately. It leads to a multiclass problem with a very large number of classes and downgrades the performance of classifiers. Actually, the outputs of different testing samples are usually interdependent. Therefore, we propose a novel method of structured classification based on SVM and hypergraph regularization (Hyper-SSVM). First, it exploits the structure and dependencies within classifying outputs. Second, we impose local constraints to samples by using Hypergraph regularization. We apply the proposed Hyper-SSVM to action classification. The experimental results demonstrate the effectiveness of the proposed method.