We derive the complete formulae governing the polarization state measurement in terahertz-time domain spectroscopy (THz-TDS) by using a rotatable THz polarizer. Four Stokes parameters can be uniquely obtained by spectrally-resolved measurement in THz-TDS. Further, we propose a new approach to measure the Mueller matrix of a pure birefringent material, using THz-TDS, by rotating the material under test. Based on the above techniques, we successfully measured the Mueller matrices of a quartz crystal in the frequency domain.