Gabor Wavelets are widely used to extract facial features since they are robust against illumination and pose changes. Due to the limitation in computational power, the common practice is to down-sample the face image to reduce number of Gabor features generated. As not all of the generated Gabor features are necessary, the main objective of this paper is to develop an efficient removal scheme of redundant filters in order to employ images with large dimension in face data processing. In particular, Genetic Algorithm is used to provide a computational and fast selection of feature ensemble. The base classifiers are trained by the AdaBoost algorithm with the Gabor feature set extracted from each single Gabor filter. By employing the joint diversity, Genetic Algorithm is then applied to select the most discriminate ensemble of classifiers followed by the optimum decision making rule on the classifiers outputs. The proposed approach is implemented in the family classification problem which has large intra-group variation. This method also allows us to select more discriminate filters from higher scales and finer orientations for those families with very young children to improve the performance with the same complexity and calculation load of the conventional Gabor wavelet set.