In this paper a novel approach is proposed for semantic video object segmentation. In particular, it is assumed that several neural network structures have been stored in a system database or memory. Each network have been learned to be appropriate to a specific application. Then, a retrieval mechanism is introduced which selects that network from the memory which better approximates the current environment. Since, however, the retrieved network does not exactly correspond to the current conditions a small adaptation of its weights will be necessary in most cases. For that reason, an efficient training algorithm has been adopted based on both the former and the current network knowledge. The former one corresponds to knowledge existing in the memory while the latter is provided by the training set selection module based on the user assistance and a color segmentation algorithm. Experimental results are presented illustrate the performance of the proposed method to real life applications.