Growing interest in power quality has led the international working groups to define new measurement techniques to apply to power systems. Special attention has been paid to harmonic and inter-harmonic measurements. This paper presents a new method to optimize harmonic estimation in power systems. The method uses filter bank property of discrete wavelet packet transform to measure the harmonics; and this is compatible with IEC standards. The method has the advantages of low spectral leakage, sub-harmonic and inter-harmonic recognition, and benefits in occasions of inappropriate length of a sampling window. The new algorithm decomposes the voltage and current signals into exclusive frequency bands corresponding to harmonics, and uses an inverse matrix optimization method to reduce the spectral leakage due to imperfect frequency response of the applied wavelet filter bank. As a case study, the method is implemented on arbitrary signals and the results are compared with the previous methods. The simulation results confirm the higher accuracy of the new algorithm.