Rotation modulation is an effective method of reducing the errors of inertial navigation system (INS). However, when the carrier is in angular motion status, especially when the heading angle changes in wide range, the performance of error suppression will be influenced. In this paper, according to the error rotation modulation principle, based on attitude error equation of rotation inertial navigation system, the mechanism that the modulation effect is influence by angular motion of carrier is analyzed. The coupling impact from the variation of carrier angle and rotation angular velocity of IMU on error rotation modulation effect is investigated. The simulation test show the result that the errors suppression performance is influenced by the variation of amplitude and rotation angle rate. And it is confirmed that low rotary rate can improve accuracy of the rotary navigation system in angular motion status of carrier.