Mortar is often viewed as a three-phase composite consisting of aggregate, bulk paste, and an interfacial transition zone (ITZ). However, this description is inconsistent with experimental findings because of the basic assumption that larger pores are only present within the ITZ. In this paper, we use backscattered electron (BSE) imaging to investigate the micro- and meso-scale structure of mortar with varying aggregate content. The results indicate that larger pores are present not only within the ITZ but also within areas far from aggregates. This phenomenon is discussed in detail based on a series of analytical calculations, such as the effective water binder ratio and the inter-aggregate spacing. We developed a modified computer model that includes a two-phase structure for bulk paste. This model interprets previous mercury intrusion porosimetry data very well.