A series of isomeric 2‐aryl‐6,6‐dimethyltetrahydro‐5‐quinolinones (set I) and 2‐aryl‐7,7‐dimethyltetrahydro‐5‐quinolinones (set II) were studied under positive ion electron ionization (EI) and electrospray ionization (ESI) techniques. Under EI conditions, the molecular ions were found to be less stable in set I isomers, and they resulted in abundant fragment ions, i.e., [M–CH3]+, [M–CO]+., [M–HCO]+, [M–(CH3,CO)]+, and [M–(CH3,CH2O)]+, when compared with set II isomers. In addition, the set I isomers showed specific fragment ions corresponding to [M–OH]+ and [M–OCH3]+. The retro‐Diels‐Alder (RDA) product ion was always higher in set II isomers. The ESI mass spectra produced [M + H]+ ions, and their decomposition showed favorable loss of CH3۬ radical, CH4 and C2H6 molecules in set I isomers. The set II isomers, however, showed predominant RDA product ions, and specific loss of H2O. The selectivity in EI and ESI was attributed to the instability of set I isomers by the presence of a gem‐dimethyl group at the α‐position, and it was supported by the data from model compounds without a gem‐dimethyl group. Density functional theory (DFT) calculations successfully corroborated the fragmentation pathways for diagnostic ions. This study revealed the effect of a gem‐dimethyl group located at the α‐position to the carbonyl having aromatic/unsaturated carbon on the other side of the carbonyl group. Copyright © 2011 John Wiley & Sons, Ltd.