In this paper, a method is proposed to solve the displacements and distortions, which are caused by inaccurate calibration in the low-level fusion. Compared with existing methods, the proposed method does not rely on any specified environmental feature and can be applied to a variety of scenarios. To implement it, twice clustering processes are applied to segment the input point cloud, and an iterative closest point (ICP) algorithm is used to iterate and correct the corresponding partitions. Furthermore, we also quantify an index to evaluate the result of correction and provide some simplified constraints to improve the measurement accuracy. Finally, the effectiveness of the proposed methods is verified by the result of 3D reconstruction.