The performance of the three-axis magnetometer is mainly restricted by the sensitivity error, offset error, and nonorthogonality error. In this paper, we present a geometric ellipsoid fitting method to calibrate these errors. The new method is based on minimizing the orthogonal distances from measuring points to the ellipsoid. The minimizing problem is formulated as the nonlinear least square problem, the problem is solved by Gauss-Newton method. The simulation and experiment results show the performance of the three-axis magnetometer is improved dramatically with the proposed calibration method.