Computer vision coupled to deep learning is a promising technique with multiple applications in the industry. In this work, the potential of this technique has been assessed in the classification of two varieties of almond trees (Prunus dulcis), Soleta and Pentacebas. For that, a convolutional neural network named VGG16 was used. The most appropriate configuration for model training was studied, which included the comparison between two different filling modes (reflect and nearest) in the data augmentation step, the evaluation of the batch size and the analysis of the image sizes. The robustness of the model was also checked, and information was obtained about how the model extracts the information from the images. The results showed that the reflect fill mode was more effective than the nearest one. The best results were obtained using batches with 30 and 40 images, with an image size of (224 × 224) pixels. The verification of the robustness proved the capability of the technique as a promising tool for plant varietal identification.