Hyperbranched polymers have shown to toughen epoxy resin with a good overall performance. In this article, a series of carboxyl group-terminated hyperbranched polyesters (HBPE-COOHs) with different backbone structures are synthesized using a simple one-pot A2+B3 approach, and subsequently incorporated into epoxy/anhydride curing system. It is found that the terminal carboxyl groups of HBPE-COOHs can promote the curing process of epoxy/anhydride system, which has a great effect on the final properties of HBPE-COOH modified epoxy thermosets. Owing to the miscible backbone structure and terminal carboxyl groups, HBPE-COOHs can efficiently toughen epoxy/anhydride thermosets with a non-phase-separated morphology. In the meantime, a simultaneous enhancement in both the elongation at break and tensile strength is achieved. The effects of HBPE-COOH structure and HBPE-COOH loading on the thermal, and thermomechanical properties are investigated. The results show that thermoset modified with HBPE-COOH3, which has the most flexible backbone structure and moderate terminal carboxyl groups, exhibits the highest crosslinking density, glass-transition temperature and on-set decomposition temperature. The toughening and reinforcing mechanism is also studied, which should be owned to the synergistic effect of chemical structure and increase in the crosslink density and fractional free volume.