To reduce handover overhead such as signaling overhead and latency, optimal handover algorithm is needed for mobile multi-hop relay (MMR) systems. This article proposes a novel handover method that reduces inter-cell handover but increases intra-cell handover events, because inter-cell handover has more overhead in view of the system and gives more latency time to users rather compared to intra-cell handover. Therefore, in the proposed scheme, intra-cell handover is encouraged instead inter-cell handover using adaptive hysteresis level. Intra-cell handover performs based on higher hysteresis level than inter-cell handover. This paper also presents hierarchical relay station ID (RS ID) concept for IEEE 802.16j standard. And, additional method using MOB_NBR-ADV message is proposed in case that hierarchical ID could not be used. This proposed algorithm can decrease handover signaling overhead, latency, and unnecessary handovers called ping-pong effect as well. Also, the proposed algorithm is backward compatible with mobile station (MS). Performance analysis and simulation validate the efficiency of the proposed algorithm.