This paper investigates the problem of joint user association and resource allocation in a downlink cognitive radio (CR) network. In the CR network, multiple cognitive base stations (CBSs) and multiple secondary users (SUs) are considered to share multiple channels with the primary user (PU) under the condition that the interference from the CR network to the PU is restricted. Exclusive user association, where each SU is allowed to associate with only one CBS, and exclusive channel allocation, where each channel is allowed to be allocated to only one SU, are adopted. Furthermore, the maximum number of SUs that can be supported by the CBS is restricted in order to prevent from CBS overloading. Under the above setups, a heuristic joint user association and resource allocation scheme is proposed. The aim is to maximize the sum rate of the SUs subject to the transmit power constraint and the interference power constraint. It is shown by simulations that the proposed scheme achieves significantly higher sum rate of the SUs than that of the fixed scheme, especially for high transmit power limit and high interference power limit.