Lots of technologies have been developed for image processing. How to adapt such technologies in video analysis to achieve a balance between accuracy and computational complexity of frame analysis for video is necessary to be studied carefully. In this paper, we propose such a framework for soccer video frames. In this framework, we first predefined 7 classes, each of which is assigned a Semantic Descriptor (SD) to delineate its semantic meaning; then two-level analyses, local-level (called block-level) and frame-level, are done to each P frame to classify it into one of 7 categories so that the corresponding SD of a category indicates the semantic meaning of the P frame. The results, which is 80.8% in average, shows the robustness of the proposed method in accuracy; Combining with the processing speed, which is 21frames per second, shows that this framework researches the balance and is very promising.