Moving object detection with moving camera is a difficult and hot issue. In order to detect moving object effectively and rapidly, this paper proposes a moving object detection algorithm by flow vector classification and multi-view geometric constraints. First, corner feature points with large eigenvalue are searched, and the feature points of present frame is matched with the previous one to compute the fundamental matrix of two images with pairs of points. From geometric aspect, the points which are far from epipolar lines are thought to be moving points. Second, due to the great different vector mode between the static points and the moving points, a flow vector classification method is adopted to lower the errors separated by geometric method. Third, removing the noise points, the moving points detected by epipolar lines and the flow vector classification determine the moving area. Experimental results show that the algorithm is accurate and real-time, processing a frame in 1ms, meeting to the real-time detection of moving object.