This paper presents a system for real-time tracking of athletes in diving training scenes. The system employs a cascade framework, and only certain regions of interest (ROI) where the athletes could appear are processed. First, background subtraction method is performed in current ROI to extract foreground regions; then, non-human foreground objects such as the springboards are excluded using prior information; next the ROI is modified so that in next frame the athlete would still be in it; finally, a fast curve-fitting algorithm is proposed to get the contour of the athlete. The curve-fitting algorithm is not sensitive to initial curve position and is able to overcome small gaps, which exist commonly in the foreground regions. The system runs at 25 HZ for 720 x 576 color images on a P4 3.0 GHZ PC