Matching people across non-overlapping camera views, a.k.a. the person re-identification problem, is important for video surveillance and gaining increasing attention. In this paper, we propose a re-identification method that uses Nonlinear Ranking with Difference Vectors (NRDV). Instead of trying to eliminate the differences between cameras or seek more reliable features, our strategy is to make full use of the targets’ differences to build a binary classifier. We then achieve re-identification through a ranking approach by employing a support vector machine with a nonlinear kernel based on radial basis function. We also propose to pre-cluster the training images using the affinity propagation clustering algorithm, and select representative images to form negative training instances. In this strategy, the classifier maintains its performance with fewer training samples, and has lower memory requirements. Extensive experiments are conducted on three public benchmark datasets, and the results demonstrate the state-of-the-art performance of the proposed method.