This paper investigates the stability and shape evolution of voids and channels under the combined effects of surface misfit, surface energy and surface diffusion. A dynamic model that incorporates the competition among these energetic forces is developed. Our approach integrates a novel local semi-implicit level set method to capture interface movement and an iterative spectral method to calculate the elastic field, which allows simulating very large shape evolution such as void breakup or coalescence in a wide range of materials systems. Our study reveals the important effect of surface misfit and remarkably rich dynamics during shape evolution. It is shown that surface misfit can lead to instabilities of voids, break-up of channels and ordering of voids.