In the recent past, deployment of Convolutional Neural Networks (CNN) has led to prodigious success in many pattern recognition tasks. This is mainly due to the very nature of CNN, that is its ability to work in a similar manner to that of the visual system of the human brain. One of the most exciting application of pattern recognition which is the focal point of the proposed work is face recognition. Despite huge efforts from the researchers spanning several decades, it has remained a substantial challenge due to inter-class and intra-class variabilities. This paper presents two simple, yet effective Convolutional Neural Network (CNN) architectures for facial image recognition. Both the architectures utilize widely adopted filters, Gabor and Frangi for facial image representation. Further, benchmark datasets such as CMU, Grimace, Yale, Face 95 and FIE have been used for testing the efficacy and an accuracy of 952% is obtained. Another aspect of the work proposed is investigating the impact of varying the number and size of feature maps. Thus, the models deployed in the proposed work are capable of exploiting spatially localized correlations in a facial image to produce consistently high accuracy.