The oil-paper compound insulation plays a vital role in the insulation structure of power transformer. To obtain the characteristics of the transformer's insulation system, it is of great importance to study the dielectric response of oil-paper compound insulation. In this study, fractional calculus is applied to model the oil-paper compound insulation. Both low-frequency and broadband high-frequency models are proposed and then they are verified by fitting the measured data of different insulation papers. Finally they are applied to two different cases: (i) the low-frequency model for fitting polarity reversal property; (ii) the broadband high-frequency model for studying the influence of the parameters variations on the frequency domain spectroscopy (FDS). The results demonstrate the advantages of the proposed models when compared with traditional ones.