Capsule endoscopy (CE) is a rather new imaging technique designed specially for small intestine that is untouchable for traditional endoscopy such as gastroscope and colonoscopy. At present, reviewing a whole CE video for each patient is an intensive task for physicians. Hence, computerized methods for a CE video is desired to reduce the review time for clinicians. In this paper, we utilize color textural features and random forests and ferns to classify CE images. A novel color uniform local binary pattern (CULBP) algorithm is first proposed, which integrates color norm patterns and color angle patterns. The CULBP feature is robust to variation of illumination and discriminative for classification. Furthermore, in order to obtain a high classification performance and efficiency, two recent machine learning approaches, i.e., random forests and ferns, are used for classification. The experiments demonstrate a very encouraging detection accuracy of the scheme.