In this paper, we firstly design a novel modulation format named Twisted-PAM4 (T-PAM4) for the high-speed NG-EPON and the PON-based digital fronthaul system. Compared to other high-order modulation formats like 4-ary pulse amplitude modulation (PAM4), T-PAM4 provides two independent sampling values in each symbol period and enables two-dimensional symbol decision, which significantly improves the receiver sensitivity and hence enlarges the system power budget. Based on this advantage, six types of T-PAM4 are experimentally demonstrated in 28-Gb/s/ λ NG-PON and 32-Gb/s/ λ PON-based digital fronthaul system. The experimental results show that, compared to the ordinary PAM4, the T-PAM4 can bring 5-dB and 4-dB extra power budget in NG-EPON and PON-based fronthaul system respectively. In addition, by experiment, it is also depicted that the T-PAM4 is remarkably robust to system nonlinearity, which would ease the linearity requirement on system components and hence reduce system cost.