In recent years, obstacle detection has been achieved using laser rangefinders or the Kinect sensor. The point cloud contains the point data of a known object such as a hand-arm robot. In this paper, to remove the known object from the point cloud, we propose a removal method that compares the depth buffer between virtual space and real space. Then a nearest neighbor search with a kd-tree is performed. Also, the closest point pair is calculated with the GJK algorithm between the components of the hand-arm robot using the structural model of the known object. Using both distances, we also propose a controller based on a control performance index containing collision avoidance information. The collision avoidance method is implemented into the slave hand-arm robot for the teleoperated system. Finally, the effectiveness of the proposed method is demonstrated experimentally.