In this paper, the output tracking problem of gear transmission servo (GTS) systems with backlash nonlinearity is studied. A new continuously backlash description-"differentiable dead zone model" is established to overcome the no differentiable property of backlash nonlinearity. First, the state space description of GTS systems with backlash nonlinearity is presented with the new dead zone model and transformed to a linear system by state feedback linearization technique. Then a nonlinear state feedback control algorithm is developed for the transformed linear system, which is proved can track globally asymptotically the smoothly desired trajectory and the limit cycles that appear due to backlash can be avoided. Simulation results validate the effectiveness of the proposed controller and the new dead zone model. The technique proposed in this paper makes backlash nonlinearity control problem be a general one which is not only to avoid the establishment of a complex multi-model switching system and inverse model compensation, but also easy to implement and expand in the engineering practice.