In this paper, a novel closed-form expression of the intermodulation distortion (IMD) of nonlinear RF power amplifier (PA) considering input and output frequency responses is proposed. A transmitter including nonlinear PA is modeled with three-box (Wiener-Hammerstein type) model to include input and output frequency responses. To verify the closed-form expression of IMD, the analytical results are compared with the simulated results by changing frequency responses of filters in three-box model. The analytical IMDs show exact agreement with the simulated ones. The proposed analytical description can be applied to accurate PA modeling and predistortion design.