Calcium (Ca2+) flux into the matrix is tightly controlled by the mitochondrial Ca2+ uniporter (MCU) due to vital roles in cell death and bioenergetics. However, the precise atomic mechanisms of MCU regulation remain unclear. Here, we solved the crystal structure of the N-terminal matrix domain of human MCU, revealing a β-grasp-like fold with a cluster of negatively charged residues that interacts with divalent cations. Binding of Ca2+ or Mg2+ destabilizes and shifts the self-association equilibrium of the domain toward monomer. Mutational disruption of the acidic face weakens oligomerization of the isolated matrix domain and full-length human protein similar to cation binding and markedly decreases MCU activity. Moreover, mitochondrial Mg2+ loading or blockade of mitochondrial Ca2+ extrusion suppresses MCU Ca2+-uptake rates. Collectively, our data reveal that the β-grasp-like matrix region harbors an MCU-regulating acidic patch that inhibits human MCU activity in response to Mg2+ and Ca2+ binding.