Since gastroscopy is able to observe the interior of gastrointestinal tract directly, it has been widely used for gastrointestinal examination. But it is hard for clinicians to accurately detect gastrointestinal disease due to its great dependence on doctors experiences. Therefore, a computer-aided lesion detection system can offer great help for clinicians. In this paper, we propose a new scheme for endoscopy image lesion detection. A trainable feature extractor based on convolutional neural network (CNN) is utilized to get more generic features for endoscopy images. And features are fed to support vector machine (SVM) to enhance the generalization ability. Experiments show that the proposed scheme outperforms the previous conventional methods based on color and texture features.