This paper presents an improved Appearance-based Average Face Difference (AAFD) scheme for face gender recognition with a low resolution and non-align thumbnail image. The main problem of the former appearance-based approaches is that not all the information is equally important in a face sub-window. Some regions in a face sub-window may have similar feature for both male and female, and some regions may contains hair, background, or noises. Thus, this work exploits the proposed face gender mask to determine the key areas in a face to classify its gender. As documented in the experimental results, with the examination of color Feret database, this method has shown that it is an effective candidate in improving the training time, recognition accuracy rate, and efficiency of overall system process for face gender recognition applications.