The improved moving object detection and shadow removing algorithms for video surveillance are presented in this paper. The proposed algorithm processes two foregrounds performed by improved GMM and chromaticity-gradient background subtraction methods. The proposed algorithm improves the classic Gaussian Mixture Model to remove some unfavorable influences, such as sudden and gradual illumination changes, ghost. In order to achieve the target, the GMM model learning rate is updated according to the illumination changes factor while the mean and variance of all distributions which match the new point are updated with different rate, and related detection mechanism is established to detect whether the slowly moving object integrate into the background. Moreover, a new effective shadow removing algorithm based on the detection results of the improved GMM and chromaticity-gradient background subtraction methods is proposed, which does not rely on color features but the contour of the moving objects to detect shadow. Finally, the related experimental results show that the improved algorithm performs more robustly and powerfully than the classical Gaussian Mixture Model in moving objects detection, and the shadow can be suppressed by the proposed shadow removing algorithm efficiently.