RFID identification has been widely adopted in access control. This kind of card or tag based approaches has a major drawback that anyone could get access with the card. In this work, we propose a neural network based face recognition system as the second access control to make sure the person granted access matches the ID on the RFID card. In this preliminary work, the face of accessing person is detected in video stream and we extract the Scale Invariant Feature Transform (SIFT) features from a face image. To enhance the generalization capability of the face recognition, we introduced the Localized Generalization Error Model (L-GEM) to train the Radial Basis Function Neural Network (RBFNN) for face recognition. Experimental results show that the proposed method could identify person that matches the RFID access card or not in a high probability.