In the high‐power five‐level neutral point clamping/H‐bridge (NPC/H) inverter, the traditional space vector modulation (SVM) algorithm at a low switching frequency has the disadvantages of a high‐voltage jump at high modulation index and an inherent narrow pulse at low modulation index, which may cause serious damage to the power stage. In this paper, a novel SVM algorithm is proposed to suppress the high‐voltage jump and narrow pulse for the five‐level NPC/H inverter with a high‐power integrated gate‐commutated thyristor (IGCT). An equivalent three‐level NPC model is applied to reduce the redundant switching state. To suppress the high‐voltage jump at high modulation index, the switching state optimization method is proposed. Moreover, the zero sequence voltage injection is proposed to suppress the narrow pulse at low modulation index. Finally, the novel SVM algorithm is experimentally tested on an experimental prototype and an engineering prototype for 6.6 kV to 20 MW motor drive system. The simulation and experimental results verify the effectiveness of the proposed SVM algorithm. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.