In this paper, a visual tracking algorithm based on adaptive background modeling and improved particle filter was proposed, which can be effectively used to track the players in sports video. Adaptive background modeling was used to extract motion information in the dynamic scene, and obtained precise motion region through a designed target selection filter, then, do the tracking algorithm under particle filter framework. The proposed algorithm solved the problem which needs to manually initialize the target position when tracking algorithm is applied to the engineering project. Besides, traditional resampling algorithm has degeneracy problem which could influence tracking effect when particle set is large, the paper also improved the resampling algorithm with a classified resample method. In the end, experimental results showed that the proposed algorithm has advantages in precision and robustness compared with the traditional particle filter.