We address the capacity of a discrete-time memoryless Gaussian channel, where the channel state information (CSI) is neither available at the transmitter nor at the receiver. The optimal capacity- achieving input distribution at low signal-to-noise ratio (SNR) is precisely characterized, and the exact capacity of a non-coherent channel is derived. The derived relations allow to better understanding the capacity of non-coherent channels at low SNR. Then, we compute the non- coherence penalty and give a more precise characterization of the sub- linear term in SNR. Finally, in order to get more insight on how the optimal input varies with SNR, upper and lower bounds on the non-zero mass point location of the capacity-achieving input are given.