This study presents a real-time highlight extraction system, based on a model-indexing decision approach, to detect and classify highlight events in baseball games. The system has several stages: caption extraction, caption identification, content recognition and model-indexing decision. A superimposed caption in the baseball videos is extracted using a multi-frame averaging technique. The caption is classified once it is extracted. The corresponding caption model, including the size and position of the individual caption data, is adopted to segment the caption content accurately into individual data. Based on the change status of the caption content, a novel model-indexing decision approach is proposed to detect and classify the highlight events of the baseball videos. Experimental results demonstrate that the proposed approach is very efficient for classifying highlight events. Furthermore, the performance of the proposed approach can be improved by 9.5% over the rule-based approach.