The transmission scheme of multilayered space-time trellis codes (ML-STTC) achieves tradeoff between diversity gain and multiplexing gain by combining space-time trellis code (STTC) and layered space-time (LST) structure in multiple-input and multiple-output (MIMO) systems. In this paper, iterative minimum mean square error (MMSE) group detection and iterative MMSE interference cancellation (IC) group detection schemes are proposed to improve the performance of ML-STTC. Moreover, the iteration can reduce the influence of the detection order on performance, and compensate for the performance loss of the firstly detected layer instead of power allocation in the iterative MMSE IC group detection scheme.