The p-type high insulating thin films were obtained by doping Mg into β-Ga2O3. Thin films with various Mg concentrations were deposited on (0001) c-plane Al2O3 substrate by radio frequency magnetron sputtering followed by post-annealing treatment. The crystal structure expanded due to the substitution of the trivalent Ga3+ with the divalent Mg2+ in a larger ion radius. The Fermi level (EF) of the Mg doped film is closer to the valence band, exhibiting a characteristic of weak p-type. The Mg doped Ga2O3 thin films were used to construct the metal/semiconductor/metal (MSM) structure, and the devices showed a high resistance (4.1pA at 10V), a high sensitivity (8.7×105%), a high responsivity (23.8mA/W) and a short decay time (0.02s) under 254nm UV light irradiation, suggesting a potential application in solar-blind photodetector.