This paper proposes a noble avoidance control algorithm based on omni-directional visual simultaneous localization and mapping (OVSLAM) with a fisheye lens camera. In addition, a robot avoids colliding with an obstacle regardless of the obstacle’s state by analyzing the information of the object obtained from an OVSLAM approach. OVSLAM has many advantages for object detection and mapping because it can measure all information around a robot simultaneously. We therefore proposed an improved avoidance and formation control to configure a multi-robot system optimized for OVSLAM. This system creates a global map based on vector information and position information of objects obtained from a local map, and determines the avoidance method according to the type of object, which is classified by analyzing the odometry and vector and position information. We carried out a formation control experiment in an environment with static obstacles and a dynamic robot, and a formation control experiment in an environment with dynamic obstacles and a robot. The reliability of the proposed formation algorithm was verified through a comparison of maps based on the proposed algorithm and real maps while maintaining the formation by applying a real robot.