Quest for ultra-low power ADCs have forced researchers to push existing ADC architectures to work in ultra-low voltage supply. VCO based ADCs are promising solutions to address this problem since continuous scaling in CMOS technology aids their highly digital nature and time-based architectures. However, VCO-based ADCs have various challenges for achieving ultra-low power targets such as small voltage margin, noise, oversampling ratio and linearity. This paper analyzes the above challenges in ultra-low voltage operation. Various existing VCO-based ADCs are introspected and the limitation at ultra-low voltage operation is discussed.