Passenger traffic has been widely neglected when considering the elevator energy consumption. Earlier, the consumption has been typically estimated in the design phase of the building by a simple energy per square meter approach. Recently, energy efficiency classification schemes for elevators, VDI 4707-1 and ISO 25745-2, have presented computational methods applying reference power measurements from the installation site to determine specific energy usage categories and to provide an estimate of annual consumption. The ratio of standby and running energy demand are based on the building type related to a certain number of starts per day. The usage of the elevator directly derives from the passenger traffic profile. Therefore, this paper highlights the importance of analyzing the passenger flow, car loading in each direction, length of trips, and their impact on the elevator energy consumption. The modeled results of the case building imply that a substantial share of the total electricity consumption is related to other forms than transporting the passengers between floors. The main cause is considered to be the low average loading in contrast to a commonly used heavy counterbalance. Thus, the paper suggests measures to decrease the energy consumption by resizing the counterbalance and compares the results to regenerative solutions. When the counterbalance is sized optimally, the modeled efficiency improvement of the counterbalance resizing nearly equals the regenerative system with approximately 60% total savings in the case office building.