Spatio-temporal differentiation method is one of the most effective methods of measuring the velocity of target. However, this method has a problem that large false velocity along the edge is often observed in the region near edge and across the boundary between objects of different velocity. This may be because the luminance in such regions changes steeply in a particular direction. To solve this problem we propose a novel algorithm that discriminates such regions and a novel velocity estimation scheme that replaces false estimate with the edge normal velocity as a second best policy. We expect that the edge normal velocity is another estimate which approximates the real velocity better. The edge normal velocity is the velocity component orthogonal to the edge purposely no matter which direction the edge actually moves in. In order to discriminate edges or boundaries, we paid attention to the ratio of eigenvalues and the third element of eigenvectors of differential coefficient matrices. We examined three types of images, which are synthesized image, captured image and image having realistic noise. Experimental results show that most of edges and boundaries are discriminated by proposed algorithm and the estimate which is better agrees with the real velocity can be obtained.