This paper clarifies the effect of color on the impression of a character. In experiments, we used four letters (B, H, N, O) as the characters. These characters are considered to represent groups categorized by shape. Black, blue, green, red, and yellow were the colors tested. The colored characters were evaluated through 24 impression word pairs. The results showed a significant difference between men's and women's impressions. It was also shown that the impression of characters can be expressed by three factors, Elegance, Liveliness, and Potency, for both men and women. The importance of the factors differed for the men's impression when compared with the women's impression. The first factor of the women's impression was Elegance, whereas the first factor of the men's impression was Liveliness. It was also shown that the color is more important than the shape of characters. The impression of black and yellow in Liveliness and green in Elegance varied for women, and the impression of blue in Elegance varied for men.