The radiative-condensation instability (RCI) in self-gravitating strongly coupled dusty plasma (SCDP) is investigated considering the effects of dust thermal velocity and polarization force on the massive dust particulates. In particular, the outer core of the dense neutron star which is supposed to be strongly coupled in nature with temperature T∼107 K and number density n∼1.3×1030 cm−3 is analyzed. The modified generalized hydrodynamic (GH) equations and electron temperature perturbation equation with radiative effects are solved using the linear perturbation method. In the classical hydrodynamic limit, the modified condition of Jeans instability owing to radiative condensation, polarization force and dust thermal velocity is obtained. In the kinetic limit, velocity of compressional mode also modifies the condition of Jeans instability. The dust thermal velocity and viscoelastic effects have stabilizing whereas polarization force and radiative cooling have destabilizing influence on the growth rate of the Jeans instability. The radiative effects stabilize the growth rate of unstable radiative modes. In isobaric mode (short wavelength), the basic condition of radiative instability is obtained which is unaffected due to the presence of polarization force and viscoelastic effects. The radiative cooling time in the outer core of neutron star is estimated and compared with the gravitational free fall time, and it is found that the cooling takes place too fast for self-gravity to be important.