We have experimentally studied the structural and elastic characteristics of rare-earth cobaltites RBaCo4–x M x O7 (R = Dy–Er, Yb, Y), in which cobalt ions are partly substituted by diamagnetic Al or Zn ions. It was found that small substitution of Co3+ ions by Al3+ ions in the YbRBaCo4–x M x O7 system (x = 0.1, 0.2, 0.5) leads to a rapid decrease and smearing of ΔE(T)/E 0 anomalies of the Young’s modulus in the region of the structural phase transition, which is accompanied by increasing hysteresis. Pure rare-earth cobaltites RBaCo4O7 (R = Dy–Er, Y) exhibit a correlation between the room-temperature structure distortion and hysteresis on the ΔE(T)/E 0 curve in a temperature interval of 80–280 K. In Zn-substituted cobaltites RBaCoZn3O7, both the hysteresis and ΔE(T)/E 0 anomalies disappear, as do low-temperature sound absorption maxima. This behavior is evidence of the suppression of structural and magnetic phase transitions and the retention of only short-range correlations of the order parameter in Zn-substituted samples.