The heterodyne laser interferometer has been widely used in ultra-precision measurement fields. However, the periodic nonlinearity error caused by frequency mixing restricts the precise measurements at nanometer level. We use a nonlinearity compensation method based on the recursive weighted least square (RWLS) algorithm. Using the compensation method, we can obtain the optimal compensation parameters of nonlinearity of heterodyne laser interferometer. To apply the RWLS, the measurement process of the laser interferometer is modeled as a state equation. The effectiveness of the RWLS approach is verified through the comparison with the reference values in some simulations.