We present a computer vision based approach for classifying graphical document images by matching distinct visual patterns present in them. To accomplish this task the image is first decomposed into congruous segments, some of which contain distinct patterns followed by image matching to identify the presence of a specific pattern in the image. We have used clustering based image segmentation to extract distinctive patterns and PCA-SIFT image features for robust image matching. We have used R-Tree based feature indexing for faster retrieval of images. We have done our experiments on advertisement images which contain company's trademarks and finally classify them based on their advertiser.