This paper describes a relative orbit estimation method for the close distance flyby adjacent to small bodies such as asteroids, comets, and natural satellites. Previous studies showed that, in a high-speed relative velocity flyby or a close distance flyby of small astronomical bodies, typical estimation algorithms such as a linear batch processing and an extended Kalman filter are no longer applicable. A new method, which only contains linear calculations, is derived considering the noise property of the line of sight vector observed by a single camera. To apply a least square method, normality of inside the square is required for an evaluation function. Mathematical formulations and numerical simulation results prove the normality to assure that the estimation is almost maximum likelihood. Numerical simulations indicate two-dimensional estimation validity of the new method. Compared with typical estimation algorithms, one-order closer estimation of small astronomical bodies can be achieved.