In hyperspectral image processing technologies, anomaly detection is a valuable and practical way of searching small unknown targets based on spectral characteristics. For the lack of prior knowledge of targets, background modeling on hyperspectral images is the key process that affects the outcome of anomaly detection operator. In this paper, a novel method of anomaly detection based on quadratic modeling is proposed. The innovation of the proposed algorithm is that it divides the detection process into two main steps: one is initial detection, which provides a preliminary judgment of background pixels; the other is the quadratic background modeling to reduce the contamination of outliers, consisting both anomaly pixels and abnormal background pixels. In the part of experiments, a semisimulated hyperspectral image and a real hyperspectral image are both used to evaluate the performance of our proposed method. Visual analysis and quantative analysis of receiver operating characteristic (ROC) curves both show that our algorithm performs better when compared with other classic approaches and state-of-the-art approaches.