The main objective in content-based image retrieval is to find images similar to a query image in an image collection. Matching using descriptors computed from regions centered at local invariant interest points (key points) have become popular because of their robustness to changes in viewpoint and occlusion. However, local descriptor matching can produce many false matches. RANSAC can robustly fit a model to data in presence of outliers and has been used to find correspondences in presence of noise. But obtaining a good hypothesis may require many runs, particularly when the proportion of inliers in the data is low. In this paper, we utilize topological information from the Delaunay triangulation to construct a refined set of matches that is presented to the RANSAC algorithm to fit a homography. Experiments show that with this refined match list, RANSAC is able to obtain correct hypothesis in very few runs. The result is often superior to ordinary RANSAC even after thousands of runs and the method consumes substantially less processing time.