This paper investigates fault-tolerant consensus scheme of leader-follower multi-agent systems when some follower agents are subject to actuator faults. Each follower agent is regarded as an independent model and relevant observer is established to accomplish a task of fault detection, based on its nominal dynamical behavior. A time-varying threshold and its corresponding condition about fault detection are obtained, which ensures that no fault alarm occurs before the threshold is satisfied. Moreover, a fault function with observer error is used to estimate the fault in the system, and an adaptive observer-based consensus protocol is designed such that all the agents can reach consensus even faults occur in the system. Finally, a numerical example is given to illustrate the effectiveness of the obtained results.