Least Square Support Vector Machines (in short LS-SVM) reduces the complexity of standard SVM to O(n2). Both SVM and LS-SVM are not suitable for the large scale regression problem. This paper proposes a modifies LS-SVM based on increment datasets, all samples' knowledge is accumulated and some samples is discarded effectively in the incremental learning process. The numerical experiments on benchmark datasets show that the proposed algorithm is considerably faster than the standard SVM and the classical incremental algorithm.