A novel phase field grain growth model combined with a micro-elasticity effect including elastic anisotropy and inhomogeneity is developed to investigate the effect of micro-elasticity on the grain growth. Due to the elastic anisotropy of each grain and the elastic interaction with its neighbors, an external elastic load affects strongly grain growth and texture evolution. A strong 〈100〉//ND fiber texture developed in poly-crystalline Cu with an initial random texture by biaxial external strain, whereas a 〈111〉//ND fiber texture evolved under the biaxial external stress condition. In contrast to the previously reported macro-elastic descriptions, there was a strong localization of strain energy density and inhomogeneous distribution even inside grains, and the elastically soft grains with a higher strain energy density grew at the expense of the elastically hard grains to reduce the total strain energy. The results suggest the possibility of tailoring the texture by controlling various external loads.