Considering unstable oscillation in the process of rotating machinery caused by changing in load lubrication and gear meshing stiffness in shaft system, the coupled nonlinear dynamic equation of rotating mechanical drive system is established with time-varying stiffness and nonlinear friction force based on Lagrange theory. The average equation of system is solved with the aid of method of multiple scales. Besides, according to Hopf bifurcation theory the stability of system is analyzed, the necessary and sufficient condition of Hopf bifurcation and periodic motion's stability are given, and the influence of supercritical and subcritical bifurcation on torsional vibration of rotating mechanical drive system is analyzed under the condition of parametric resonance and internal resonance. Last, the numerical simulation verifies the results. A significant contribution of this study is to ensure smooth running of rotating machinery system, and provide a theoretical basis for the future design of mechanical component.