The investigation of the CaBaCo4−xFex07 oxide shows that a huge decrease of ferrimagnetism is produced for very low Fe contents (x < 0.05) at the benefit of magnetic phase separation and magnetic frustration. This gigantic effect of iron doping at the Co3+ sites in CaBaCo4O7 is compared to that produced by diamagnetic Ga3+ and Al3+ dopants, showing that the antiferromagnetic interactions between the dopant Fe3+ and the Co2+ ferromagnetic chains has a crucial role in the destabilization of the ferrimagnetic structure, whereas the triangular geometry of the cobalt sublattice imposes the appearance of magnetic frustration.