In this paper, we build a convolutional neural network for gender classification based on facial image. And we take experiments with AR face database. The network is built up with an input layer, two convolutional layers, two down-sampling layers and a full-connected layer. In the experiments, we achieve 92% classification accuracy. We also test it with image rotated 15 degree at most, the average accuracy can achieve 91.6%. When occlusion is more than 20%, the misclassification rate raises obviously.