As generating the wideband modulation signals, the signal quality is usually degraded by many impairments such as I/Q path delay, non-flat RF frequency response and etc. In this paper, the authors propose two methods by using Keysight SystemVue™ to create the modulation signals and download them to an arbitrary waveform generator (AWG) for further calibration. To compare these two methods, a 64 QAM modulation waveform with a bandwidth over 1GHz was evaluated. The first method is multi-tone scalar calibration which improves the error vector magnitude (EVM) from 5.3 % to 4.2 %. The other one is vector calibration which improves the EVM from 5.3% to 1.1%. Finally, these signals are used to measure the system performance of an amplifier. As found, the modulation signal with vector calibration can show the more exact effect of AM/AM and AM/PM on EVM of the amplifier than scalar does.