To identify the optimal duplex mode and maximize the offered gain in the relay system, we compare the performances of half-duplex relay (HDR) and full-duplex relay (FDR). Finite-state Markov chain (FSMC) models are presented, showing the joint effects of finite-length queues and adaptive modulation (AM) under uncorrelated Rayleigh fading channels. Transmission efficiency, packet loss rate, and delay are derived and compared based on these models. According to our results, FDR is superior to HDR in the low-signal-to-noise ratio (SNR) region but loses its advantage as the SNR increases. When the size of the queue is large enough, FDR can be superior to HDR, even in the high-SNR region.