Recent research has thrown considerable light on the history of the domestic sheep, but has not extended to ancient sheep specimens. In the present study, ancient DNA analysis was carried out on eight archaeological sheep remains recovered from Erlitou archaeological site in Henan Province (ca. 2100–1800 B.C.) to explore the genetic structure of ancient sheep and the phylogenetic relationship between ancient and modern sheep. We analyzed the control region sequences and coding regions of mitochondrial DNA from the remains by direct sequencing and restriction fragment length polymorphism analysis, respectively. Our results reveal that all ancient sheep belong to lineage A defined by modern sheep sequences. Phylogenetic analysis shows that neither argali (Ovis ammon) nor urial (Ovis vignei) mtDNA is closely related to Erlitou ancient sheep. In addition, our results suggest that ancient DNA analysis can serve as a powerful tool in tracing prehistoric population movement.