In this paper, a novel two-stage noise removal algorithm to deal with impulse noise is proposed. In the first stage, an adaptive two-level feedforward neural network (NN) with a backpropagation training algorithm was applied to remove the noise cleanly and keep the uncorrupted information well. In the second stage, the fuzzy decision rules inspired by the human visual system (HVS) are proposed to classify the image pixels into human perception sensitive class and nonsensitive class, and to compensate the blur of the edge and the destruction caused by the median filter. An NN is proposed to enhance the sensitive regions with higher visual quality. According to the experimental results, the proposed method is superior to conventional methods in perceptual image quality as well as the clarity and smoothness in edge regions.