To discover novel heterocyclic compounds with high herbicidal activity, a series of new pyrimidyl‐1,2,4‐triazole derivatives (Ι‐1 to Ι‐41) was designed and synthesized by incorporating the activating 1,2,4‐triazole, pyrimidine, and phenyl sulfonyl structural templets. Their structures were clearly identified by proton nuclear magnetic resonance (1H NMR), carbon‐13 nuclear magnetic resonance (13C NMR), high‐resolution mass spectroscopy (HRMS), and X‐ray single‐crystal diffraction (compound Ι‐3). Their herbicidal activities against monocotyledons (Echinochloa crusgalli, Sorghum bicolor) and dicotyledons (Raphanus sativus, Brassica campestris, Cucumis sativus, and Medicago sativa) were evaluated using a Petri dish culture method. The results indicated that most of title compounds exhibited significant herbicidal potencies against the plant growth at 100 mg·L−1, while some of them, such as Ι‐2 and Ι‐4, showed remarkable inhibitory effects on weed control at 10 mg·L−1 and 100 mg·L−1. On the whole, compound Ι‐3, which was found to possess an even broader spectrum of inhibitory activities against the roots and stalks of Echinochloa crusgalli, Raphanus sativus, Brassica campestris, Cucumis sativus, and Medicago sativa at 10 and 100 mg·L−1 than that of the commercial herbicide Flumetsulam, could be employed as a new lead structure for further optimization to find more potent herbicides.