The changes in structural, magnetic and calorimetric properties induced by B substitution of P or Si in MnFe0.95P2/3Si1/3 are reported. The evolution of the structure and in particular of the ratio in cell parameters c/a is specific to the B substitution. The disappearance of the first-order magnetic transition (FOMT) with B substitution is shown to be systematic and much more pronounced than observed when varying the Mn/Fe and P/Si ratios or using other non-metallic elements. The correlation between the evolutions of the discontinuities Δ(c/a) and ΔStr suggests that the weakening of the FOMT results from a modification of the magneto-elastic coupling. The evolution seems in line with theoretical studies on Fe2P1-xXx materials and points towards a combined effect of size differences and valence changes at the non-metallic site.