This paper presents a novel vision servoing approach using depth maps to perform robotic motion task with field of view (FOV) constraint. The vision servoing scheme relies on the depth information available from an Red Green Blue-Depth (RGB-D) camera. With respect to the previous approaches, the proposed vision servoing approach has the advantage as follow: First, it does not require the estimation of 3D pose, which only relies on the error of depth maps, without feature matching. Moreover, this method addresses the problem of field of view constraint in the motion process by the frame pose estimation. This visual servoing system is computation efficiency, because it does not need the descriptor and estimation of the pose parameters. We drive the robot towards the target position by the depth map error minimizing. Both simulation and experimental results are provided to demonstrate the effectiveness of the proposed depth map-based vision servoing method.