First of all, mathematically detailed non-linear and linear relative dynamic models for satellites in formation flying in low earth orbit (LEO) are derived and next state estimation based on Kalman filter is emphasized. The Extended Kalman filter (EKF) and linear Kalman filter (LKF) for nonlinear and linear relative models respectively are compared in this paper. By increasing relative distance between satellites in formation, EKF based on nonlinear model outperforms the LKF which is based on linear model. A comparison of the centralized EKF and the centralized LKF has been made to show that centralized EKF exhibits better performance for large formations in terms of absolute estimation error.