Radiation detectors based on diamond are highly favored for particle physics research due to the superior radiation hardness. In this work, we investigate the influence of impurities and crystalline imperfections on the charge collection efficiency (CCE) of single crystal diamond. Seventeen (17) ultra-low fluorescent diamond samples grown by microwave plasma chemical vapor deposition method from IIa Technologies PTE LTD are pre-selected for this study. The measured CCE of all samples using 241Am (α-particles) as ionizing source are analyzed together with the concentration of trace impurities and crystalline imperfection in the crystal. The amounts of impurities are quantified from integrated fluorescence intensity arising from the nitrogen vacancies (NV) created during different CVD growth process conditions. The crystal imperfections are assessed by X-ray rocking curves from X-ray topography images obtained at the Cornell High Energy Synchrotron Source. The CCE decays rapidly as the intensity of NV (INV), phonon sideband approaches that of diamond 2nd order Raman peak which follows the relation: CCEINV=100/1+INV1.052. The energy resolution, ΔE/E (ratio of the energy spectrum width to the most probable peak) highly correlates with broader rocking curve width distribution.This work provides an understanding on the most important factors that contribute to degradation of charge collection efficiency (CCE) in diamond based detectors and sensors. The CCE decays rapidly as the intensity of nitrogen vacancy phonon sideband approaches that of diamond 2nd order Raman peak which follows the relation: CCEINV=100/1+INV1.052. The energy resolution, that is the ratio of the energy spectrum width to the most probable peak, highly correlates with broader X-ray rocking curve width distribution.