This paper suggests a thin cloud removing approach of color remote sensing image based on support vector machine in HSI color model. The intensity component is decomposed in multi-scale by using strong edge capture ability of support vector machine, and the coefficients in different scales are obtained. Then abundant high frequency information is obtained by combining with directional filter bank. Reconstructed image is obtained through enhancing coefficients of high frequency and suppressing coefficients of low frequency. The saturation component is processed by exponential expand method, while the hue component is invariant. Experiments show that thin cloud can be removed efficiently by using the method introduced in this paper.