In this letter, we describe a method to estimate the time delay for speaker localization in reverberant environments. Based on an adaptive eigenvalue decomposition (AED) algorithm, the method takes the reverberation fully into account by estimating channel impulse responses from a speaker to sensors directly, but it may suffer from whitening effects for temporally correlated natural sounds. Imposing sparse priors on the responses can reduce the temporal whitening and provide a more accurate and robust time delay of speaker location. Experiments demonstrate that the proposed method can efficiently estimate the time delay for speaker localization in reverberant environments.