Realizing both ultralow breakdown voltage and ultrahigh gain is one of the major challenges in the development of high‐performance avalanche photodetector. Here, it is reported that an ultrahigh avalanche gain of 3 × 105 can be realized in the graphite/InSe Schottky photodetector at a breakdown voltage down to 5.5 V. Remarkably, the threshold breakdown voltage can be further reduced down to 1.8 V by raising the operating temperature, approaching the theoretical limit of 1.5 /e, with the bandgap of semiconductor. A 2D impact ionization model is developed and it is uncovered that observation of high gain at low breakdown voltage arises from reduced dimensionality of electron–phonon scattering in the layered InSe flake. These findings open up a promising avenue for developing novel weak‐light detectors with low energy consumption and high sensitivity.