The tumor targeted fluorescent magnetic IR780-Fe3O4 nanoparticles were prepared, for the separation and detection of CTCs. The synthesis of IR780-Fe3O4 nanoparticles were characterized by electron microscopy, fluorescence spectrometer, and superconducting quantum interferometer. Then the effect of IR780-Fe3O4 nanoparticles on the tumor and normal cells were analyzed by confocal microscope and flow cytometry, and the confocal microscope was also used to target the location of IR780-Fe3O4 nanoparticles in MCF-7 cells. The standard curve was drawn and evaluated according to the IR780-Fe3O4 nanoparticles fluorescence intensity of tumor cells after incubation. The results showed that IR780-Fe3O4 nanoparticles could target a variety of CTCs, furthermore, the cellular localization experiment proved that IR780-Fe3O4 nanoparticles could target at the mitochondria of tumor cells. The established method of coupling magnetic Fe3O4 nanoparticles IR780 could well distinguish the tumor cells and normal cells, and also could be used for separating and detecting CTCs in simulated of blood samples.