In current study the influence of weight percentage of high impact polystyrene (HIPS), nanosilica and also hardener content on damping mode properties of epoxy/HIPS/SiO2 hybrid nanocomposite has been investigated. Mixture design has been employed to present mathematical models for predicting first and second damping mode of new mentioned hybrid nanocomposite as function of physical factors and optimizing mentioned mechanical properties. Totally 10 experiments were generated according to simplex centroid mixture design and 10 samples were prepared and tested based on portions of mixture components. The maximum and minimum values of first mode of damping occurred in run numbers 10 and 1 that were 3.74% and 1.71% respectively, also the maximum and minimum value of second mode of damping occurred in coded levels 6 and 1 with the values of 4.61% and 1.02% respectively. Results of analysis of variance showed that all of selected input variables had linear effect on both of the studied responses, two component interactions among HIPS and hardener (X1∗X3) and interaction among silica and hardener (X2∗X3) affected damping 1st and 2nd modes while interaction among HIPS and silica (X1∗X2) only affected on the second mode of damping and did not have any clear effect on damping 1st mode due to its P-value. Moreover optimization was done using response optimizer part of Minitab software and the values of target and allowable minimum responses were defined for the software and finally it calculated the maximum achievable values for both of the responses and also corresponding coded values of mixture components. Optimization results described that the highest value for damping 1st and 2nd modes were 3.80% and 4.65% respectively with coded values of HIPS=0, SiO2=0.37 and hardener=0.62 and corresponding mixture components were HIPS=2wt%, SiO2=4.33wt% and hardener=27.1phr respectively.