The paper presents a comparative investigations of the state estimation of a power system with a phase shifting transformer in the polar and rectangular coordinate systems. The estimation with the weighted-least-squares method using the augment matrix approach is considered. During the investigations, one performs of analysis of impact of a coordinate system, in which the state vector is considered, and data redundancy on such parameters of a state estimation as: the number of iterations, a condition number of a coefficient matrix, parameters characterizing accuracy of calculation results.