This paper studies the problem of fault estimation of time-varying delay systems using adaptive fault diagnosis observer (AFDO). Based on the proposed fast adaptive fault estimation (FAFE) algorithm, which does not need the derivative of the output with respect to time, a delay-dependent criteria is first established to reduce the conservatism of design procedure, and the FAFE algorithm can enhance the performance of fault estimation. In terms of linear matrix inequality (LMI), we derive sufficient conditions for the existence of AFDO. Finally, simulation results are presented to illustrate the effectiveness of the proposed algorithm.