We assessed the efficacy and safety of sitagliptin compared with α‐glucosidase inhibitor (αGI) in 120 of Japanese patients with type 2 diabetes mellitus (T2DM) inadequately controlled on stable ≤2 mg/day glimepiride alone [mean hemoglobin A1c (HbA1c) 7.7%] by the randomized, active‐controlled, non‐inferiority trial. Patients were randomly assigned to receive additional sitagliptin or αGI for 24 weeks. The primary endpoint was change in HbA1c from baseline to week 12. After 12 weeks, sitagliptin reduced HbA1c by −0.44% (p < 0.001) relative to αGI. At 24 weeks, the reduction was almost identical between the groups (−0.091%, p = 0.47). Gastrointestinal disorders were more common with αGI than with sitagliptin, but only minor hypoglycaemia occurred in both groups at similar frequency. These data suggested that sitagliptin was not inferior to αGI for reduction of HbA1c in Japanese T2DM patients receiving glimepiride alone, and well tolerated with minimum risk of gastrointestinal symptoms and hypoglycaemia.