Esophageal cancer is one of the most common malignant tumors in Henan Province and videoendoscope is an invaluable tool for the diagnosis of early esophageal cancer. Therefore, we propose a diagnosis expert system for the diagnosis of early esophageal cancer on the basis of videoendoscope. We apply the Bag-of-Words model to get the feature representations and group the features using the Information Bottleneck method. Then we build the rule database by the aid of expert-doctors. Finally, we use the Petri net as the inference engine. Real Application shows that the accuracy of the proposed expert system is over 95%.