Parallel phase-shifting digital holography (PPSDH) is a technique of single-shot phase-shifting digital holography. We found that there are two problems with this technique. (1) Some extraneous noises caused by the intensity unevenness of the reference wave become slightly superimposed on the object image. (2) The conjugate image cannot be completely removed. This is because the object wave causes the phase-shift error by illuminating an image sensor with a large incident angle. To solve these problems, we propose an algorithm for removing residual 0th-order diffraction and conjugate images in PPSDH. In the proposed algorithm, we modified phase-shifting interferometry in order to work through the unevenness of the intensity distribution and applied the Fourier transform technique to PPSDH to remove the residual conjugate image. The effectiveness of the proposed algorithm was experimentally verified.