Visual Object tracking is one of the key problems is machine vision. A contourlet transform based visual object tracking method is given in this paper. Being one of the multi-scale geometric analysis methods, the contourlet transform has better performance than the traditional wavelet transform to describe the edge feature of visual objects. The contourlet coefficients are firstly obtained and parts of them are then used to calculate the “edge” energy of visual objects. The tracking of an interested object is then done according to a diamond search method, which finds a square region with a similar energy to the object in the successive frames of the video. Experimental results show that the proposed method is effective and efficient.