This paper is concerned with the hypercoercivity property of solutions to the Cauchy problem on the linear Boltzmann equation with a confining potential force. We obtain the exponential time rate of solutions converging to the steady state under some conditions on both initial data and the potential function. Specifically, initial data is properly chosen such that the conservation laws of mass, total energy and possible partial angular momentums are satisfied for all nonnegative time, and a large class of potentials including some polynomials are allowed. The result also extends the case of parabolic forces considered in Duan (Nonlinearity 24(8):2165–2189 (2011)) to the non-parabolic general case here.