Digital images are usually represented in the form of a matrix that indicates the data of intensity of each pixel. When the data of intensity of each pixel are considered as the data of the height in the coordinates of the pixel and the pixels are smoothly connected, the digital image can be reconstructed as a curved surface. Interpolation using a polyharmonic function can interpolate a curved surface smoothly using the boundary geometry of the region and arbitrary internal points in the region. In this study, we applied the interpolation method to image processing, and propose a method which can restore and enlarge images smoothly. The method proposed here can realize the simultaneous restoration and smooth enlargement of images, which has been difficult to achieve by conventional methods. This method is applied to the enlargement of images while restoring damaged areas or to the enlargement of an image of the face of persons who is wearing glasses while removing the glasses frame from the image, and its effectiveness is demonstrated. In addition, a method of increasing processing speed by division processing is also examined, and its validity is confirmed. © 2010 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.