We present a method of face recognition using facial texture and surface information. We first use Gabor wavelets extracting local features at different scales and orientations by gray facial images, then combine the texture with the surface feature vectors based on principal component analysis (PCA) to obtain feature vectors from gray and facial surface images. We propose a hybrid Taguchi particle swarm optimization (HTPSO) algorithm for face recognition based on multilayer neural networks as an identification model. Experimental results demonstrate the efficiency of our method for different face poses and facial expressions. In addition, our work compared with other proposed approaches such as back-propagation (BP), particle swarm optimization (PSO) and the genetic algorithm (GA). With different data modality the experimental results demonstrated that the proposed HTPSO learning algorithm is better than the other approaches in recognition rates. The texture and shape features can improve the efficiency of face recognition.