Energy harvesting devices based on the piezoelectric effect that converts ambient energy to electric energy is a very attractive energy source for remote sensors and embedded devices. Although topology optimization has been applied to the design of piezoelectric transducers, the locations of piezoelectric materials are predefined and only the optimal layout of elastic materials is considered. In this paper, both elastic materials as well as piezoelectric materials are considered for the design of energy harvesting devices under the topology optimization formulation. The objective function for this study is to maximize the energy conversion factor. The sensitivities of both stored strain energy and electrical energy are derived by the adjoint method. Examples of energy harvesting devices are presented and discussed using the proposed method.