Delay phenomenon commonly exists in most load balancing systems for parallel computing. It can cause some unstable oscillatory actions and intensely affect the performance of the load balancing system. In this case, a time delay feedback control model is presented to describe the dynamic load balancing system in parallel surrounding. By choosing proper Lyapunov-Krasovskii functionals and using Moon inequality and Schur complement lemma, the optimal control law is obtained not only for the different delay conditions but also in a scalable system scale. At last, the simulation experiments based on multi-threads proved the validity of the theory and method introduced.