In this paper a novel bi-directional diffusion model with two layer constraint is proposed. The new model can smooth the artifacts and jaggies in the interpolated image and make image clearly by enhancing the edge. It considers both along and orthogonal to edge diffusion, and the orthogonal to edge backward diffusion enhances the edge in image. There is severe block effect in the interpolated image, if the orthogonal to edge diffusion is directly adopted in the interpolated image, jaggies will be strengthened. With the two layer constraint, the gradient magnitude of original un-interpolated image is mapped to the interpolated image as a sign matrix. The selection of diffusion direction is controlled by the mapped sign matrix; only the edge pixel in original image is enhanced when interpolating. Using the novel model proposed in this paper, processed image is clearer than that processed by the along edge diffusion model; with the two layer constraint, jaggies are diminished and the image is smoothed