Genetic diversity grows with the population size in most neutral evolution models. Empirical evidence of large populations with limited diversity has been proposed to be due either to genetic bottlenecks or to selection. An alternative explanation is that the limited diversity is a result of rare reproduction events. Indeed, recent estimates of the offspring number distribution highlight the role of large reproduction events. We here show that in a large class of models containing such rare events, genetic diversity decreases as the population size increases, in neutral evolution models. For many realistic offspring number distributions, the contribution of rare events to the dynamics grows with the population size. In the context of genetic diversity, these rare events induce a decrease in the time to the most recent common ancestor and in the genetic heterogeneity as the population grows. This phenomenon may explain the observed rapid fixation of genes in large populations, in the absence of observable selection or bottlenecks.