In paper the design sensitivity analysis of dynamic characteristics of structures with viscoelastic (VE) dampers is considered. The dampers are modeled using the classical rheological model and the rheological model with fractional derivatives. The design sensitivity is analyzed by using direct differentiation method and adjoint variables method. The formulae enabling the calculation of sensitivity of the first and the second order with respect to chosen design parameters have been derived. Sensitivity analysis of eigenvalues, eigenvectors and dynamic characteristics such as natural frequencies and non-dimensional damping ratios are presented. The correctness of the presented formulae is illustrated in examples.