Rice panicle blast grading is very important in gauging cultivar resistance and in the precise control of a blast epidemic. However, the development of an automated, rapid, and accurate panicle blast grading system is a challenging task. This is mainly because of the complexity of the pathology, appearance, and definition of the blast disease level. In this study, a new method for grading panicle blast based on hyperspectral imaging technology is proposed. The method is based on the concept of the “bag of textons,” which is widely used in the document analysis field and which defines a “bag of spectra words” (BoSW) model for hyperspectral image data representation. Hyperspectral image data representation based on the BoSW model is used as the input to a chi-square kernel support vector machine (chi-SVM) classifier for predicting the rice panicle blast level. Specifically, the BoSW model jointly considers the image-spectrum information to attain improved classification accuracy. It reduces a high-dimension hyperspectral image into a compact, low-dimension representation. It also highlights the histogram statistics of a typical spectrum prototype to reflect the panicle blast severity level. In this way, it avoids the fine segmentation and morphological analysis of blast lesions. Experiments were conducted on a total of 312 fresh rice panicles covering more than 50 cultivars, which were collected from an experimental field under natural conditions. The results showed that the proposed method could effectively grade panicle blast with classification accuracies of up to 81.41% for six-class grading and 96.40% for two-class grading in the validation datasets. Comparison experiments were conducted on different data batches or combinations thereof. The results showed that the technique is viable for different types of rice cultivar and planting seasons, pointing to its widespread practical applicability.