This paper presents the application of magnetic encoders in the actuator servo system based on FPGA for the unmanned helicopter. The magnetic encoder outputs two quadrature voltage signals which change according to the angle position. The digital decoding circuit is designed based on the vector rotation algorithm in the controller, which can provide a digital angular position with 16 bit resolution. By building the digital decoding circuit in the controller, hardware sources in FPGA are fully utilized and the reliability of the system is improved furthest.