In this paper, we formulate image annotation as a semi-supervised learning problem under multi-instance learning framework. A novel graph based semi-supervised learning approach to image annotation using multiple instances is presented, which extends the conventional semi-supervised learning to multi-instance setting by introducing the adaptive geometric relationship between two bags of instances. The experiments over Corel images have shown that this approach outperforms other methods and is effective for image annotation.