It is very important to understand quantitatively the relationship between the difference of an audio signal from the external noise and the psychological impression of the noise, to design a comfortable sound environment. In this paper, a method for predicting and/or estimating the human psychological response to the external noise in the case of listening to the audio signal, is proposed by using the fuzzy set theory. Concretely, when the persons listening to the audio signal are exposed to meaningless random noise, bi-variate membership functions, the supports of which are the spectral distance measure and signal-to-noise ratio, are first established. Next, a method for evaluating the psychological response is proposed by introducing the concept of the fuzzy probability. Finally, the validity and the applicability of the proposed method are confirmed experimentally by applying to the actually observed data. The experimental results are in good agreement with the theory.