For the state consensus problems of high-order linear multi-agent systems with time-varying delays in directed topologies, the LMI based consensus criterion and NLMI based protocol design are investigated. Improved Lyapunov Krasovskii functional is used for establishing the consensus criteria and deriving the corresponding consensus protocol. In order to reduce the conservativeness, some proper Free weighting matrices are added into the derivative of Lyapunov Krasovskii functional, and that only keeps one necessary zoom. The numerical and simulation examples are given to demonstrate the effectiveness of the theoretical results. Compared with existing literature, the convergence criterion and protocol design proposed have lower conservativeness.