Polynomial chaos theory can be used to approximate a stochastic linear dynamical system by a deterministic linear system of larger dimension. This approximation enables the use of linear control theory tools for the analysis and design of controllers for the original stochastic system. The approximate system can, however, have stable moments while the moments of the original stochastic system are unstable, which poses an important theoretical challenge. This article presents error bounds for the first- and second-order moments of linear stochastic systems that are obtained via polynomial chaos expansions. These bounds can be used to design controllers that are robust to the approximation error, which is demonstrated in some illustrative simulation results.