Identifying moving objects from a video sequence is a fundamental and critical task in many computer vision applications. We propose a three stage adaptive object segmentation algorithm for color surveillance videos. In the first stage, background is modeled using Multiple Correlation Coefficient (Ra,bc) using pixel-level based approach for motion segmentation. Segmented foreground objects generally include their self shadows as foreground objects since the shadow intensity differs and gradually changes from the background in a video sequence. To eliminate such shadows from motion segmented video sequences in the second stage, we propose an algorithm based on inferential statistical Difference in Mean (Z) method. Cast shadows produce troublesome effects for video surveillance systems, typically for object tracking from a fixed viewpoint. To eliminate cast shadows from video sequences in the third stage, we propose an algorithm based on the fact that, cast shadow points are usually adjacent to object points and are merged in a single blob on the edge of the moving objects. The approach uses the Chi-Square Distribution (χ) to build statistical model. χ models are constructed and updated for every inputted frame. Results obtained with different indoor and outdoor sequences show the robustness of the object segmentation approach.