We report a real-time experimental demonstration of a long-reach coherent ultra-dense wavelength-division multiplexing (WDM)-passive optical network (PON) system with quadratic-amplitude modulation (QAM)-based dual-polarization optical transceivers. Using field-programmable gate array-based digital signal processing, the transmission of 20-WDM channels in 2.5-GHz grid, employing both 2.5-Gb/s dual-polarization quadrature phase-shift keying (DP-QPSK) and 3.75-Gb/s DP-8PSK signals, is successfully demonstrated with bit-error rate in real-time. The performance is evaluated over 100-km standard single mode fiber in terms of receiver sensitivity and ODN power budget for both modulation formats, showing the feasibility of coherent PON scenarios using flexible dual-polarization strategies supported by software-defined transceivers.