Machined surface topography and surface roughness are significant factors that directly affect the service performance of the material. A simulation of surface topography in ball-end milling of AISI H13 steel is developed based on the relative motion between cutting tool and workpiece. The developed model was verified by milling experiment and can be used to simulate the machined surface topography accurately. In order to optimize the surface roughness by taking material removal rate (MRR) into account, a simulation trial is conducted by employing the product value (p) and ratio (r) of feed per tooth fz and radial depth of cut ae based on the developed model. The effect of r and p on three dimensional (3D) arithmetic average deviation Sba has been investigated. An optimizing model for Sba with regard to p and r is developed. For a given value of p, which means for constant MRR, the value of r for minimum Sba can be calculated. The validation of the optimizing model was conducted by experiment, and the model was proved to be able to precisely predict Sba within the range of cutting parameters in this research.