Seismic hazard analysis and vulnerability evaluation are two parts of seismic risk assessment. The form of the vulnerability evaluation procedure in HAZUS99, which is widely adopted in US and world wide, is taken as reference. A hybrid approach integrated the Simplex with the Generic Algorithm is adopted to inverse the parameters of capacity curve and fragility curve from the earthquake damage matrix of local buildings. A ground motion parameter based vulnerability evaluation approach is developed. Thus, it is easy to calculate building vulnerability for a given demand spectra. To illustrate and validate the approach, all parameters are inversed from a typical vulnerability matrix of multi-story masonry building.