Feature point matching method combining color information has being attracting more and more attention in vision navigation because it is one of the most important aspect, and the matching result which only using the gray or RGB color space information of image can't satisfy the requirement. The paper put forward a new method which utilized the HSV (Hue, Saturation, and Value) color space information of images to describe the feature points, and also proposed a new method to detect the mismatched points with the output of inertial navigation system. The principal component analysis (PCA) was used to reduce the dimension of the point describing vectors. The experiments validated the proposed methods, and proved that feature representations combining all the three color information is more effective than only one.