Millimeter wave compact RF transmitter and receiver for 5G communication system are designed in this paper, which could be typically applied in massive MIMO system of 16 channels. The transmitter and receiver are designed using multilayered structure and highly-integrated chips to decrease the module size. The antenna feeding transition structure are designed using electromagnetic field simulation software, and integrated in RF module. The size of fabricated RF modules is 2cm–2cm. IF and power supply circuits are also designed separately for both of them to support the RF array with 16 channels configuration. Performance of single channel transmitter and receiver are measured which could be applied in Q-band millimeter wave massive MIMO system.