In this paper, a stochastic optimization algorithm is proposed to calibrate the three-axis magnetometer onboard. The sensor errors, namely, hard iron, soft iron, nonorthogonality, scale factors, and bias, are taken into account. Particle swarm optimization (PSO) strategy is used to do the calibration, enhanced by the stretching technique to improve the accuracy and robustness. The performance of this algorithm is evaluated with a series of laboratory experiments and a field experiment of autonomous underwater vehicle. Comparisons with other analytical calibration methods are made. The results demonstrate that both the PSO and the stretched PSO algorithm can significantly compensate the magnetometer readings, and the latter algorithm has higher accuracy and more robustness.