In this paper, we present a new approach for tracking targets with their size and shape time-varying, based on a combination of mean-shift and affine structure. Although the well-known mean-shift colour-based tracking algorithm is an effective tracking tool, difficulties arise when it is applied to track a size-changing visual target due to the fixed kernel-bandwidth. To improve this, the present study employs a corner detector on the object candidate from mean-shift and reconstructs the target position and relative scale between frames using the affine structure available from two or three views. In comparison experiments against previous algorithms, the present model shows better tracking-consistency and good efficiency. Our algorithm is also demonstrated in a real-time implement controlling the pan-tilt-zoom parameters of an active camera. The results indicate the modelpsilas tracking capability in the presence of scale change and partial occlusions.