Craters are the most abundant landform on the planet surface, which could provide fundamental clues for planetary science. Due to variations in the terrain, illumination, and scale, it is challenging to detect craters through remote sensing images and it requires an effective crater feature extraction method. In this paper, we address this problem using Gist features, which can provide highly effective descriptions on crater’s local edges and global structure. The proposed crater detection procedure contains three key steps. First, we extract all candidate craters on a planet image using a boundary-based technique. Second, Gist features are generated from selected training samples. Third, crater detection is conducted using Gist feature vectors with random forest classification. Compared to pixel-based and Haar-like features, our method shows more accurate crater recognition, and achieves satisfied results in the experiments conducted on the Mars Orbiter Camera (MOC) database.