With the increasing number of shops and service providers offering loyalty cards and attractive benefits for card holders, the amount of different cards a typical customer owns has become a problem. In most cases, a card needs to be scanned in the particular shop to gain discounts meaning the customers have to always carry a large number of cards. In this paper, we present an approach to automatically detect the card provider, card type and customer number, which are then stored within a mobile application, allowing customers to carry all their cards with them on their smartphone. We propose ACC∗ as a new evaluation metric and evaluate the performance of this approach on a large real-world dataset of loyalty cards.