In this paper, a motor-generator (MG) control algorithm was proposed to improve the shift quality for a dual mode power split transmission (PST) hybrid electric vehicle (HEV). In order to analyze the shift characteristics during the mode shift, Bondgraph models for transient-state were constructed and state equations were derived. From the state equations, it was found that inertia torque of MG2 causes a transient torque during the mode shift. Based on the transient torque, a MG torque control algorithm which compensates the transient torque was developed to reduce the driveshaft torque variation. To evaluate the performance of the MG2 control algorithm, dynamic models of the dual mode PST HEV were obtained and an AMESim-MATLAB/Simulink based mode shift performance simulator was developed. It was found from the simulation results that the driveshaft torque variation was reduced by the proposed control algorithm which provides improved shift quality.