Image based localization has been developed for many applications such as mobile localization, auto-navigation, augmented reality and photo tourism. When the querying image is matched against a pre-built 3D feature point cloud, its pose can be estimated for future use. However, when the querying image is distant from the pre-built 3D point cloud, conventional single image-based localization method will fail. To address this problem, we present an incremental image set querying based localization framework. When single image localization fails, the system will incrementally ask the user to input more auxiliary images until the localization is successful and stable. The main idea is that image set, instead of single image, is matched against the pre-built 3D point cloud to meet the challenge. Next the image set is incrementally enlarged and aggregated to form a local 3D model. Compared with single image querying based localization method, the querying 3D model contains more information and geometry constraints which are essential for localization. Experiments have demonstrated the effectiveness and feasibility of the proposed framework.