Correct assessment of image fidelity is fundamental to the development of efficient image compression schemes especially to that of the schemes taking characteristics of the human visual system into account. In this paper, a fidelity metric for assessing the visual quality of color images is presented. The proposed fidelity metric is designed to measure the perceivable distortion in the quasi-uniform color space of CIELAB. To evaluate the perceptibility of distortion, a color visual model is employed to estimate the visibility threshold of distortion for each color pixel. The visibility threshold of each color is measured as the size of the sphere of just-noticeable color difference and is modeled as a function of chroma, local luminance gradient and background uniformity. Simulation results show that the assessment of the proposed fidelity metric is more correspondent with the subjective assessment than other metrics using PSNR and CIELAB DeltaE.