Detection and tracking is still a difficult and important problem. In order to improve the algorithm of detection and tracking in our work, an efficient method is proposed in this paper. The main research contents are as follows: 1)moving detection of fish based on RGB-color and contour information, where multi-threshold is used to segment the color image. 2)moving tracking of fish based on interframe global nearest neighbour (IGNN) algorithm. Experimental results demonstrate that detection inaccuracy and shadow are solved. Meanwhile, the method is robust to illumination and noise, so that it can meet the requirements of actual applications.