Video target tracking is a complex task, specially when the tracking system is expected to work well in different scenarios. For this reason, this paper proposes an architecture based on a two layer image-processing modules: general tracking layer (GTL) and context layer (CL). GTL describes a generic multipurpose tracking process for video surveillance systems. CL is designed as a symbolic reasoning system that manages the symbolic interface data between GTL modules in order to assess a specific situation and take the appropriate decision. CL intervenes at three different stages of the tracking process, these are initialization, association and update. Our architecture has been tested in two different scenarios to show the advantages in improved performance and output continuity.