The purpose of this paper is to develop a wrapper Random Forest-based feature selection method and to study the performance on emotion recognition of different selected feature sets. A large bank of Gabor filters is used to extract the face appearance. A feature selection is then applied on the wide feature set based on feature importance score computed by Random Forest. A multi-class SVM is finally trained on the chosen features using a widely used database (CK+ database). Results show the impact of the chosen features on the recognition rate and reveal that anger, sadness and the neutral expression recognition is increased by feature selection.