This paper presents a method of a person following mobile robot with obstacles avoidance under varying illumination. The person following has some useful applications, such as a transport of luggage and a good walking companion. And it also has technical challenges, such as keeping the detection of a person in the daily space and moving in the disorderly environment. A detecting method of a target person in complicated sight and robust extraction of target person's color under varying illumination by using disparity image and H-S Histogram is proposed. The following motion of a mobile robot for avoiding obstacles based on a potential method is applied. The validity of the proposed method is confirmed though experiments.