Deploying ultra dense networks (UDNs) is a major trend in the evolution of cellular networks, where the number of base stations (BSs) may exceed the number of users. In this paper, we investigate energy and spectral efficient frequency reuse strategies in hexagonally deployed cells, where BS sleeping is allowed for a cell without active users. Toward this goal, we derive the closed form expression of the energy efficiency (EE)and spectral efficiency (SE), and find the optimal frequency reuse strategies that maximize the EE and SE, respectively. Our analyses show that full frequency reuse is SE-optimal for cellular networks with all ratios of BS to user density, but is EE-optimal only when the BS-user density ratio exceeds a threshold (i.e. in UDNs), while using orthogonal frequency band among adjacent cells is EE-optimal when the ratio of density is less than the threshold. The normalized EE gain of full frequency reuse in UDNs increases with the number of BSs. Simulation results validate our analysis and illustrate the EE gain with typical BS-user density ratios.