Scale adapting and occlusion handling are two important issues in visual tracking, but usually tricky for many algorithms to address them efficiently. In this paper, we propose a framework for multiple-obstacle tracking in maritime scenes with the aid of depth information, which renders the vital clue for scale adapting and occlusion handling. In the proposed framework, each detected obstacle is tracked by an independent tracker respectively, and the depth of an obstacle is obtained by computing the disparity between the obstacle centers in left and right rectified images. Thereafter, the bounding box scale of a tracked obstacle is adapted according to its linear relationship to the depth. When occlusion occurs, the occluded obstacle is terminated to track, and when it reappears, the same identity number before occluding is assigned to it to keep the tracking consistent. Experimental results on our own dataset demonstrate that the proposed approach performs superiorly to the state-of-the-art tracking method in terms of scale adapting and occlusion handling.