To realize the augmented reality (AR) visualization of pre- or intra-operative 3-dimensional (invisible) information about the interior of tissues such as a tumor in a 2-dimensional endoscopic view, we proposed a novel method for endoscopic image overlay called “direct calibration of endoscopic camera (DCEC)” and developed a prototype of the system based on the three-dimensional measurement using micro-fluxgate magnetic sensors in combination with the DCEC. The DCEC requires neither pre-operative camera calibration nor intra-operative endoscopic camera tracking. Using the DCEC, the world-to-endoscopic image transformation can be directly calculated in an intra-operative stage. The calibration protocol is flexible and applicable to a variety of endoscopic cameras such as oblique-viewing endoscopic cameras, and easily usable in the operating room. To evaluate the performance of the DCEC and applicability of the developed system in clinical use, we performed an in vivo experiment. As a result, our system could overlay the position of a magnet embedded in the vicinity of a target tumor in real-time.