In markerless tumor tracking using X-ray fluoroscopic images for image-guided radiation therapy (IGRT), robustness against mistracking is important. Mistracking is primarily caused by bone structures projected on fluoroscopic images because they generally hide tumor features. We devised a real-time tumor-tracking method. The novelty of this method is to control “importance recognition” in computers, i.e., controlling recognition of image features that are important or unimportant for tumor tracking. This study demonstrates the feasibility of our method using clinical X-ray fluoroscopic images.The strategy of our method was to utilize different positional co-occurrence probabilities in training images for supervised deep learning. A bone digitally reconstructed radiograph (DRR) was randomly overlapped on a soft-tissue DRR to create training images for deep learning, which established positional decorrelation between the bone feature and ground truth, indicating tumor position and shape. As the training of deep learning processed, this decorrelation induced the computer recognition that the bone feature was unimportant for tumor tracking. Moreover, this method was fully optimized for a patient because DRRs were created from patient-specific CT data only, unlike a general medical application of deep learning such as diagnosis that uses a massive amount of patient data.The accuracy of lung tumor tracking was within approximately 1 mm although the low-visible tumor and spine were overlapped on X-ray fluoroscopic images.Feasibility of robust tumor tracking by our method was demonstrated. This method can be expected as a breakthrough technique that can contribute to the robust and safe implementation of markerless tumor tracking for radiation therapy in future.