Primate immunodeficiency viruses, including HIV-1, are characterized by the presence of accessory genes such as vif, vpr, vpx, vpu, and nef. Current knowledge indicates that none of the primate lentiviral accessory proteins has enzymatic activity. Instead, these proteins interact with cellular ligands to either act as adapter molecules to redirect the normal function of host factors for virus-specific purposes or to inhibit a normal host function by mediating degradation or causing intracellular mislocalization/sequestration of the factors involved. This review aims at providing an update of our current understanding of how Vif, Vpu, and Vpx control the cellular restriction factors APOBEC3G, BST-2, and SAMHD1, respectively.