Interleave division multiple access (IDMA) is a potential candidate for the future fifth generation (5G) systems. In this paper, we propose a novel architecture for IDMA system with low latency while maintaining low complexity. In the conventional architecture, the IDMA receiver sequentially processes deinterleaving, despreading, spreading and interleaving for multiuser detection. The proposed architecture, which is called the interleaved domain, can perform multi-user detection directly without deinterleaving the received frame. Because of this, the interleaving is no longer needed in the interference cancellation loop resulting in the decrease of latency by half and the increase of throughput by twice. In VLSI implementation results, the proposed architecture has reduced circuit area and power consumption by 53% and 58% compared to the conventional architecture on the same throughput condition.