Currently much of the pre-harvest fruit valuation is still done by farmers or technicians that visually inspect the pieces of fruit. However, this process has great limitations since their decisions have high subjectivity and a thorough analysis of the whole production, or even a significant part of it, is unapproachable. Therefore, computer vision and machine learning techniques are increasingly being introduced into this process. In this work, we deal with the problem of automatically identifying plum varieties at early maturity stages, which is even difficult for the human expert. To face that identification, we propose a two-step procedure. Firstly, captured images are processed to identify the region where the plum appears. Secondly, we determine the plum variety using a deep convolutional neural network. Experimental results show that the proposed system achieves a remarkable behavior, with accuracy values that range from 91 to 97%.