In this paper, a manipulation classification method based on SURF features was proposed for near-duplicate videos (NDVs). After detect corresponding keyframes based on Zernike moments, we use the SURF matching algorithm to find matched point pairs. The orientation among these corresponding point pairs is measured to evaluate the consistency for manipulation classification. The experimental results show that the proposed method can not only raise the performance of near-duplicate video retrieval (NDVR) but also determine the type of manipulation for NDVs well.