There are some problems in the video identity recognition, such as low efficiency of data processing and instability of the identification performance. This paper designs specific links and the corresponding processes for the interested character recognition and scene tracking, which are including the construction of adaptive segmentation, video face flexible recognition, feature library model, optimizing search strategies based on genetic algorithm, scene path adaptive generation, all these are in order to provide reference for automatic interested character recognition and scene tracking.