A novel adaptive digital predistortion method combines RLS (Recursive Least-Square) and LMS (Least Mean-Square) with Volterra kernel is proposed to compensate the nonlinear and memorial distortion of OFDM power amplifier. For RLS could induce unexpected modulation mistakes in small error signal, it is chosen to estimate coefficients in big error signals to ensure the high convergent speed. As soon as the value of error signal is less than defined error threshold, adaptive algorithm is changed from RLS to LMS automatically. LMS that has a stable convergence in small error signals can update coefficients in a smooth convergent process. This particular characteristic is also beneficial to hardware implementation to trace the time variability of power amplifier and compensate memorial nonlinear effects. Furthermore, the constellation distortion and spectrum characteristics of 16-QAM modulated OFDM signals with different predistortion algorithms are analyzed in detail by the simulation process. Compared with conventional single RLS or LMS predistortion algorithm, the results show that the performance of digital adaptive predistortion system could be optimized with both high speed and stability by this new method.