Aiming at the image retrieval technology based on color feature and shape feature, a new image retrieval algorithm based on color segment histogram and Hu moment invariants with edge information was presented. The idea of color segment was to divide image space into fixed blocks, and then to set each block as different weights according to the color distribution characteristic. Moreover, image of shape feature was obtained according to the moment invariants and the edge image. Finally, user assigns the weights to the features respectively, and calculates the similarity with combined features of color and shape. Compared with the traditional single method, the results indicate that the proposed algorithm in this paper has better retrieval recall and precision.