In order to restrain the problem of low automation level of feature detector and high matching consuming as for conventional local matching algorithms, the paper proposes an optimization matching algorithm based on Harris and SIFT algorithm. In this algorithm, Harris corner detector based on the ideology image break raises automation level of feature detector, and then merging algorithm optimizes the initial feature points in order to increase matching speed firstly. It constructs the SIFT descriptor for image feature description secondly. The matching result is finally obtained by the nearest neighbor matching algorithm on the condition that feature points are well-proportioned distributing. In addition, it applies the knowledge of analytic geometry to calculate the distance between matching point and epipolar line to reduce the error matching. The experimental results prove that the combination of those algorithms is effective. This algorithm wins high matching accuracy and matching time-consuming cuts down.