When AdaBoost algorithm is used for face detection, it may easily lead to overfitting problem if training samples contain noise or are difficult to classify. In this paper, we focus on designing an algorithm named AdaBoostRF, using Random Forests as weak learners. To obtain a set of effective Random Forests weak learners, firstly, CART algorithm was used to construct the base classifier. Then the weak classifier was obtained by using simple majority voting method. Finally, the required strong classifier was obtained after T cycles. Compared with the existing AdaBoost methods, the AdaBoostRF provides a possible way to handle the overfitting problem in AdaBoost. Experimental results based on MIT-CBCL face database showed that the detection performance of the AdaBoostRF algorithm has been improved, and its overall performance is better than that of the AdaBoostSVM algorithm. Experimental based on unbalanced data sets of MIT+CMU face database showed that the overfitting problem has been improved effectively.