Phosphate rock (PR) and phosphogypsum (PG) contain heavy metals, radionuclides and are exposed to ultraviolet and cycles of desiccation (i.e., Gafsa and Sfax in Tunisia). Nevertheless, an extensive diversity of bacterial species has been identified in such extreme and nutrient-poor environments. This mini-review aims to summarize findings published so far about microbes isolated, enriched from or able to grow on PR and PG. Concerning the ecology of these two complex environments, this survey has underscored the importance of key microbial genera or consortia associated with at least one of these phenotypes: (1) phosphate solubilization, (2) heavy metals tolerance, (3) radiation resistance—positively correlated to desiccation tolerance, (4) sulfur oxidation—associated to polyphosphate stocking and (5) sulfate (SO42−) reduction.