In recent years, the research on the relative navigation of UAV formation flying goes a bit deeper. The traditional relative navigation mainly adopts the combination of GPS and INS, but it is easy to be interfered and has low accuracy, so the research on the visual and INS integrated navigation is becoming hot spot. In order to solve the navigation parameter estimation problem which is based on visual formation navigation system, the research on adopting determination sampling type filter to completing the estimation of navigation parameters is carried out. Firstly, according to motion equation of formation, massive point equation of formation, visual measurement equation which is in the wingman coordinate, this paper establishes system state equation and measurement equation which is fitted to determination sampling type filter, then modifies the non-local effect through the method of orthogonal transformation. The simulation results show that the method can meet the requirements of this paper.