A parallel distributed optimization method for the minimization of the total resource of a system with discrete elements is proposed, and theoretical and experimental verifications are carried out in this paper. The distributed optimization algorithm consists of two processes, namely the resource reduction process and the resource addition process. In the former process, each element discards its critical resource margin which is the minimum among the resource margins with respect to global and local constrainsts while in the latter process, a small amount of resources are added to all the elements. Some rules for adjusting the additional resources are introduced to obtain fast convergence and better solutions. The proposed method is sucessively applied for optimizing electric circuits and discrete structures, and the method is found to be effective, very robust and suitable for parallel processing. The proposed distributed optimization algorithm is found heuristically, but its effectiveness is also analyzed theoretically.