Cognitive radio could detect the white space of spectrum and utilize spectrum resource efficiently. In a cognitive radio system, the recognition of signal modulation is a key technology, which would help the cognitive radio system to configure and realize intelligent green communication. In general, the recognition of signal modulation is not a linear classification. Back propagation (BP) neural network could solve the nonlinear classification. In this paper, we propose a training technique, cubature Kalman filters (CKF) to train a BP network. The network could better classify the nonlinear problem for the modulation recognition in a cognitive radio system. Through the simulation, the results show that the proposed training technique works better than existing techniques for nonlinear modulation classification in a cognitive radio system.