In this paper we introduce the design and implementation of an automatic camera servo system using improved frame differential algorithm which can be used in varying environments. The system uses frame differential algorithm which is improved by us to analyse the screen shots. In order to make sure that the moving object is always in the center of the shot, the system detects the target motion and sends orders to the cradle head through the serial port to track the moving object according to the analysis results. In addition, for the sake of distinguishing people from small moving things, the size of the tracking target is limited to a certain range according to the size of lens aperture.