A vector theoretical model involving the polarization spectrum behavior of the stimulated Brillouin scattering (SBS) is proposed. It reveals a phenomenon of spectral polarization spreading, i.e., the states of polarization of different frequency components of signal light experience distinct SBS pulling. The polarization behaviors in birefringence-free single-mode fiber (SMF), practical SMF, and polarization maintaining fiber (PMF) are investigated by analytical discussion, simulation, and experiment. It is found that in arbitrary fiber, the concurrent polarization evolution induced by fiber birefringence and SBS polarization spreading can be decomposed as a linear fiber birefringence rotation, followed by the SBS polarization spreading effect. The orientation and the size of SBS polarization spreading in the free-depletion regime are explicitly determined. Moreover, an orientation switching performance of the polarization spreading is observed in PMF. Finally, we demonstrate the SBS polarization spreading in SMF and PMF experimentally, and good agreement with theory is found.