Using imperfect channel state information (CSI) may cause severe violations of the interference restriction in cognitive radio (CR). We consider designing a robust CR system, over either SISO frequency-selective or MIMO channels, with multiple primary users (PUs) and multiple noncooperative secondary users (SUs), who form an ad-hoc network that is naturally modeled as a noncooperative game. The imperfectness of PU CSI is taken into account through the worst-case robustness philosophy. We study the existence and uniqueness properties of the Nash equilibria (NE) of the robust games, and devise distributed algorithms with their convergency properties to achieve the competitive optimality for the SU network. As special cases, our framework also provides, through convex optimization, the robust power allocation and precoding for each SU.