Emotion recognition based on physiological signals has a significant future of research and applications. However, in the process of emotion recognition, it is difficult to obtain the most significant feature combinations. Dual-Structure Particle Swarm Optimization (DSPSO) is applied to select emotion features of physiological signals so as to improve the recognition rates in this paper. K-Nearest Neighbors (KNN) is applied to PSO to select optimal feature subsets. This paper proposed incremental K for avoiding indivisibility about multi-classification. In view of repeated emergence about same swarms when iteration tends to be convergent, look-up table method is presented to avoid superfluous calculation. The experiment results demonstrate that these improved methods are feasible and efficient.