Probabilistic tracking algorithms typically using linear structure to update the learning model. Such linear structure is not appropriate for long-term robust tracking as the occlusion and other challenging factors may interfere the processing of incoming frames. Recently a spatio-temporal context (STC) algorithm based on Bayesian framework has using the context information between the target and its locally contexts to help tracking. In this paper, we propose an adaptive structure model that can help to discard the negative information during the tracking. This model establishes multi-templates to hold the credible information while tracking, when one of the templates gets the better confidence coefficient, this template will replace the current template and update the learning model. Furthermore, an improved scale update scheme is proposed to handle the scale variations problems in STC. Extensive experimental results show that our tracker's superior robustness and accuracy against the original STC algorithm.