We have proposed a “zoom-in” PET system that integrates a higher-resolution detector capable of measuring depth of interaction (DOI) into an existing scanner to obtain high-resolution images of a targeted region with high-sensitivity. The system acquires coincidence events between the high-resolution detector and low-resolution detectors, as well as those between the low-resolution detectors. Because of the irregular system geometry and the use of a DOI detector, the system matrix of the zoom-in PET system requires far greater storage space than that of a conventional PET scanner, which also results in long computational time for image reconstruction. To address this issue, here we propose a system matrix factorization for the zoom-in PET to reduce the storage and computational cost while maintaining the accuracy in image reconstruction. The proposed factored system matrix consists of two major components: a detector response (or sinogram blurring) matrix and a geometrical projection matrix. We present a novel method to design the geometrical component and an iterative algorithm to estimate the detector response matrix. A 2D simulation study showed that the proposed method can reduce the storage space and reconstruction time by a factor of 5 without noticeable sacrifice in image quality.