In this paper, a facial expression recognition algorithm based on Gabor and conditional random fields is proposed. Firstly, owing to the fact that in the existing databases, the number of people and images are relatively small, we established our own facial expression database, and some preprocessing methods are performed thereon. Secondly, Gabor features are extracted in five scales and eight directions from the facial expression images, which are performed dimensionality reduction via PCA and the feature information with less dimension is acquired. Finally, facial expression is recognized and classified by using Conditional Random Fields (CRFs). In the experiments based on the facial database we established, the overall average recognition rate of the proposed algorithm is up to 91.62%, which shows the effectiveness of the proposed algorithm.