In this paper, a relational matching approach for imagery-to-GIS data is presented. This method applies image aspect interpretation and geospatial data mining techniques to realize their integration. Three-dimensional (3D) primitives, standing for house, are chosen, and their projections are represented by the aspects. The hierarchy aspect graphs are constructed to represent their connected relations. In this connection, the arcs are described by attribute data via the formulated coding regulations. The nodes of the graph represent image features and their attributes can contain measurements on these features. The arcs of the graph represent relations between features and their attributes can contain measurements on spatial relations. The data mining is used to discover the semantic relationship of these primitives. The aerial image is interpreted via these aspects and geospatial data mining. The experimental results demonstrated that the presented method is capable of effectively interpreting the aerial images and extracting the high accuracy of DBM (digital building model) at a rate of 83%.