An extremely low-complexity inter-carrier interference (ICI) suppression method for OFDM systems in the presence of phase noise is proposed in this paper. The core idea is to utilize the ISI-free part of cyclic prefix (CP), which is simply ignored in traditional methods. We linearly combine the ISI-free samples in CP with the corresponding samples in the OFDM symbol to suppress ICI, and this scheme is called `CP combining' throughout this paper. In addition, the optimum combining coefficients, in the sense of ICI power minimization, are derived, and a set of near-optimum coefficients are proposed to reduce the complexity. Simulation results show that the proposed CP combining method improves the system performance by 0.5-1.5dB. Furthermore, the proposed method can be incorporated with other phase noise mitigation methods to further improve the system performance.