Target detection has an extremely wide range of applications and great significance in civil and military fields. Nevertheless, with the increasing complexity of application environments, traditional methods cannot satisfy the requirements of the application in some aspects. In this paper, an improved infrared and visible target detection method under complicated background is proposed, which makes some innovations to traditional methods. Firstly, edge gradient information is utilized to prevent the slow-moving target or long-time stationary target from being incorporated into the background gradually. Meanwhile, HSV color model is employed to remove shadows. Moreover, in this paper, the problem of ghosting has also been improved. Finally, the strategy of discarding small targets and repairing holes is utilized to perfect the detection results. Experimental results demonstrate that the presented method can effectively improve the defects of traditional methods, and it simultaneously has better robustness for infrared or visible moving object detection under complicated background.