A cyclometalated iridium(III) complex has been synthesized as a novel luminescent probe for G-quadruplex (G4) DNA in aqueous solution, and a G4-based assay was employed for the detection of nicking endonuclease activity using Nb.BsmI as a model enzyme. In the assay, an initial hairpin DNA (5′- TG4AG3TG4AG3TG4A2G3A2TGCTA3TAGCAT2C3T2C4AC-3′) sequence containing the Nb.BsmI recognition site in its stem region and a partially caged G-rich DNA sequence at its 5′-terminus could be cleaved by Nb.BsmI, which liberates the G-rich DNA sequence and enhances the luminescence of complex 1. This assay showed a linear relation between luminescent signal and Nb.BsmI concentration over the range of 0–10U/mL (R2=0.9937) with a detection limit for Nb.BsmI of 0.042U/mL. Furthermore, this assay showed high selectivity for Nb.BsmI over other nicking endonucleases, and functioned effectively in the presence of cell lysates.