This paper proposes a self-organizing map for the visual perception of a partner robot. The robot should memorize its interacting human, and also the environment as a background, because the background restricts the perception of the robot and gives suitable information to the robot. First, the robot memorizes the background by using difference extraction of images, and color information of the image by using k-means algorithm. And then, the robot learns the topological structure of the color patterns clustered by k-means algorithm. The experimental results show that the robot can recognize the human and several postures efficiently.