A novel mixed-mode stationary phase (OAS) has been prepared by bonding n-octyl and aminopropyl groups with the surface of silica gel. The obtained OAS stationary phase was characterized by Fourier transform infrared spectroscopy, elemental analysis, zeta potential and ion-exchange capacity. The mixed-mode one-site and two-site retention models were used to quantitatively describe the retention mechanism of the OAS stationary phase using three strong acidic analytes. The results indicated that the two-site model is more suitable for the description of mechanism of OAS phase. Then the OAS stationary phase was applied to separate ten water- and fat-soluble vitamins in a single run; the separation is based on the hydrophobic and ionic interaction mechanisms. A method for the simultaneous determination of ten water- and fat-soluble vitamins in commercial multivitamin tablets by high performance liquid chromatography with ultraviolet detection was established and validated according to the stability, linearity, accuracy and precision. Finally, the validated method was used to analyze the content of ten vitamins in two commercial multivitamin tablets and the recoveries were in the range of 89.2–111.0 %, with RSDs <6.1 %, indicating the good reliability of the proposed method.