In this paper, we propose an audio speaker handover system that is designed to enable automatic selection and control of most suitable audio output devices. For smooth device handover, we also propose a modified received signal strength (RSS) based distance estimation method that limits the transmit power control function. From the experiments, we will show in this paper that the proposed audio device handover system achieves good performance in the aspects of handover delay time and handover failure rate.