The segmentation of some scenes can be better for its next step in the processing and analysis. In this paper, the Gaussian mixture model clustering is used to detect and segment the scenes in the sports competition. Firstly, the main color of the scene is extracted by the method of color space histogram, and it is used as a sample for local training. Then we use the expectation maximization algorithm to estimate the GMM local parameters, to achieve the modeling of the scene. The algorithm can solve the problem of color nonuniformity of the scene that the color feature method cannot compensate. After obtaining the candidate area of the scene, the problem of noise is solved by the linear fitting method of Hough transform and least squares method, and the edge result is refined to realize better scene image segmentation. The experimental results show that the method can find out the main colors in the scene accurately and efficiently, and it has good precision and recall rate.