Machining problems often occur when the fillet area of a part is being milled. A general geometrical model of circular milling has been established, on the basis of which the maximum radial engagement angle formula is derived. The impact of cutting conditions on the maximum radial engagement angle is investigated through simulation. As the normal cutting force has a major impact on the machining deformation, the effect of radial engagement angle, axial depth of cut and helix angle on the maximum normal cutting force is also studied based on the classic cutting force model of end milling. The analytical chatter stability model for linear milling can be applied to the circular milling; the precondition is that the maximum radial engagement angle instead of the nominal one should be used in simulation. An example of pocket milling was presented to reveal how to apply the dynamic simulation technology to the cutting conditions optimization of circular milling.