Background counts on the XRS calorimeter spectrometer of the Suzaku Observatory (formerly Astro-E2) have several sources, including primary cosmic rays and secondary particles interacting with the pixels and with the silicon structure of the array. Using ground data, we identified the types of correlations between events on different pixels, and between pixel pulses and the signal from the anti-coincidence detector behind the calorimeter. In this paper, we present details of the ground background events and the rejection criteria required to remove them while minimizing deadtime. We also present the in-orbit background measured during the time that XRS was functioning in orbit.