Fast and reliable detection of moving objects is one of the important requirements for video surveillance systems. Mean shift based non-parametric background modeling supports more sensitive and robust detection in dynamic outdoor scenes. However it is prohibitive for real-time applications such as video surveillance. This paper aims to deal with the limitation of high computational complexity. Firstly, coarse to fine method are proposed to avoid raster scanning entire image. Foreground pixels are detected in coarse level to roughly locate the foreground objects in the image, and then finer detection is performed on the corresponding blocks gradually. Secondly, fast mean shift approach is presented according to temporal dependencies. Mean shift iterations are performed starting from incoming data and the mode found in last time. The experimental results show that the proposed algorithm is effective and efficient in dynamic environment.