Nd-rich phase plays a critical role in wetting grain boundary and facilitating texture formation for hot deformed (HD) Nd-Fe-B magnets. In this study, a non-uniform distribution of Nd-rich phase with dimension up to a few micrometers was observed in nanocrystalline HD magnets. The aggregation of the Nd-rich phase is confirmed to result from the low density precursor prepared by spark plasma sintering (SPS). The large local demagnetizing fields induced by Nd-rich phase aggregation led to the open recoil loops and reduced coercivity. Upon reducing recoil loop openness by eliminating Nd-rich phase aggregation, the coercivity of the HD magnet was significantly improved from 226 kA/m to 995 kA/m, and a high maximum energy product of 293 kJ/m3 was obtained. The dependences of microstructure and coercivity on the recoil loop characteristics suggest an essential approach for improving the magnetic properties of nanocrystalline HD Nd-Fe-B magnets.