Remote sensing image registration is still a challenging task because of diverse image types and the lack of a consistent transformation. To improve image registration in remote sensing, this paper develops a robust and accurate feature point matching framework. A modified scale-invariant feature transform (SIFT) method is first introduced for feature detection and pair matching. Based on the properties of matched pairs, the standard grouping boundary (SGB) and confidence elliptical boundary (CEB) are computed for further examination. The SGB is utilized to categorize matched pairs according to the pair slopes. The CEB is further employed to remove outliers whose feature positions are outside the elliptical contour. Finally, the random sample consensus (RANSAC) approach is implemented to acquire the most appropriate transformation function for image registration. The proposed image registration algorithm has been tested on numerous multi-temporal remote sensing images. Experimental results validated the improvement in feature matching accuracy, which resulted in better registration performance over state-of-the-art methods. This new framework is of potential in many remote sensing applications that require automatic image registration.