The ultrasonic vibration (UV)-assisted method, as an innovative nanomachining process, has competitive advantages compared to traditional atomic force microscopy (AFM) nanomachining methods. However, the mechanism of nanomachining by ultrasonic-assisted AFM is still unclear. Furthermore, the mathematical control model for the nanomachining process is still lacking. Therefore, the UV-assisted nanomachining process is difficult to control on the nanometer-thick film, and no additional work has been reported at this time. In this paper, the mechanisms of ultrasonic-assisted AFM subnanomachining have been analyzed by using a point-mass model for the dynamic AFM cantilever, and a mathematical model of ultrasonic subnanomachining has been established on the basis of these mechanisms. The subnanomachining experiments were carried out on a 5-nm-thick polystyrene thin film using AFM under sample UV conditions. The experimental results have shown that the amplitude of sample UV can regulate the subnanomachining depth. Finally, the simulation results from the mathematical model and the experimental results have been compared in this study.