The deformation behaviors of AZ80 magnesium alloy were systematically investigated by the combination of simulations and experiments. Specifically, the isothermal hot compression test of AZ80 magnesium alloy was performed on Gleeble-3500 thermo-simulation machine in the temperature range of 573–723K with strain rates from 0.001 to 1s−1. It is found that the flow stress is very sensitive to deformation temperatures and strain rates. On the basis of microstructure evolution, a DRX kinetics model was proposed by using regression analysis in microstructural evolution of AZ80 alloy with a consideration of effects of dislocation density and nucleation rate on the DRX. The developed DRX model was coupled into the finite element software, DEFORM-3D, along with cellular automaton (CA) based on physical model, to simulate the microstructural evolution in the deformation zone. The developed CA model with DRX is able to predict the nucleation and growth of DRX grains (R-grains) of the testing material in hot working process. Microstructural evolution and the distribution of average grain size in final microstructure show a good agreement between the simulated results and experimental ones. It suggests that the present CA model with DRX is feasible in studying the DRX process of AZ80 alloy.