The small slope approximation (SSA) method is a practical method to calculate the electromagnetic (EM) scattering from rough surfaces. However, the SSA method requires that the interval for sampling surfaces must be small enough, such as less than one-tenth of incident wavelength. This constraint condition will cause the problem of huge memory consumption and insufficient memory when the EM scattering of large rough surfaces is calculated. Although the hard disk has large space to keep data and can solve the insufficient memory problem, its read/write speed is still too slow. In addition, massive data transmission will reduce the computational efficiency for the compute unified device architecture (CUDA) parallel computation under some conditions. In this paper, the main idea of the spectral decomposition modeling method is that the whole spectrum of rough surface is divided into several parts and these parts can be used to generate different-scale rough surfaces. Then, by analyzing the different-scale rough surfaces, the large rough surface can be achieved and applied to the calculation of EM scattering with the SSA method. Due to the small memory consumption of different-scale rough surfaces, it takes less time to translate data for the different-scale rough surfaces than that for the standard large surface. Thus, the spectral decomposition modeling method could readily be applied to CUDA parallel computation.