The quality of the olive oil is directly related to fruits status and the extraction process conditions. A correct classification of olive lots, depending on their quality, is critical to obtain the best possible oil. Currently, the separation in different categories is done manually before the oil extraction process. This article presents an on-line computer vision system which automatically classifies olive lots according to their quality level in different categories. The classification is based on the differentiation between olives that have been picked up from the ground or from the tree. Two full setups have been installed and tested in real production conditions in the reception of the mill: before and after the olive washing process. The suggested approach uses a feature vector that concatenates the olive image histograms from different colour spaces and the texture values of three algorithms (image entropy, grey level co-occurrence matrix and statistics like Contrast, Correlation, Energy and Homogeneity). As classifiers, an Artificial Neural Network (ANN) and a Support Vector Machine (SVM) were used. For the experimental validation, 6325 images from 100 batches were analysed showing good classification results (success ratios of 98.4% before the washing stage working with an SVM and 98.8% after cleaning using ANN algorithm).