To deal with the rigid template matching problem in real-world scenarios, we propose a novel iterative feature-pair updating framework which is also robust to high levels of outliers, such as background changing, complex nonrigid deformation and partial occlusion. Given a pair of template image and target image, we first extract a set of corresponding feature-pairs as candidates. Then, we propose a robust objective function under the iterative framework for discriminatively updating these candidates, where the space distance, appearance distance, and the overlapping percentage of feature pairs are integrated simultaneously. Finally, a hierarchical matching strategy is provided with the parameter discussion. Experimental results compared with the-state-of-art methods on public data sets demonstrate the effectiveness of the proposed method.