In this paper, we propose a computer vision system that detects the external defects of orange citrus fruits using multi-spectral imaging sensor. First, the proposed algorithm segments the orange fruit from the captured Near-Infra Red (NIR) and RGB images using only the NIR component. Second, some adaptive pre-processing techniques are applied on the segmented RGB and NIR orange fruit images. Hence, a thresholding technique is utilized in order to detect the defects in seven different color components of the orange fruit. Finally, a voting process is applied on the seven thresholded color components images to determine if the citrus fruit image is defected or defect free. The overall accuracy of the algorithm is more than 95%, and the proposed algorithm can process three images of resolution (640×480 pixels) per second.