In this paper, a novely method for speaker system is proposed. The least square support vector machine (LS-SVM) based on the quadratic equality constraints is analyzed firstly. A speaker recognition system is then designed based on LS-SVM. The Mel frequency ceptral coefficients (MFCCs) are adopted as the speaker speech feature parameters in the system. The MFCC feature parameters are trained and tested independently using the Gaussian radius basis function. We evaluate the LS-SVM system using the voice sets of similarly pronouncing speakers in the same recording conditions. The training time and equal error rate between LS-SVM and conventional SVM in the experiments are compared. The results show that the speaker recognition based on LS-SVM has less computational complexity, shorter training time than the speaker recognition based on the conventional SVM. In order to test the recognition system, we use the VQ and GMM model. The results show LS-SVM has high right recognition rate and adaptability for speaker recognition.