In this paper, the AP clustering algorithm is improved by defining a new similarity to be applied in the ploarimetric SAR image classification. On this basis, a new unsupervised classification method is proposed which combines the Four-component decomposition and the improved AP clustering. The proposed method mainly consists of three steps: Firstly, Four-component decomposition is adopted to produce initial segmentation. Secondly, the improved affinity propagation clustering based on the Wishart distance measure is applied on the initial segmentation to merge clusters and obtain an appropriate number of categories. Finally, an iterative algorithm based on the complex wishart density function is applied. The effectiveness of this algorithm is demonstrated by the test with NASA/JPL AIRSAR L-band data of San Francisco and Flevoland.