This paper presents a robust real-time method for general detection of abandoned and removed objects from surveillance videos. The system introduces a unique combination of a new pixel-wise static region detector and a novel abandoned/removed object classifier based on color richness. In the static region detection phase, two backgrounds are constructed respectively to build foreground and stationary masks which are then used to update a static region confidence map. Static regions are thus extracted from the confidence map and further classified into abandoned or removed items by comparing color richness between the background and current frame. Our algorithm is easy to implement, robust to small repetitive motions, illumination change and can handle object occlusion. Experimental results on two public video databases which are shot in different scenarios demonstrate the robustness and practicability of the proposed method in real-time video surveillance.