Let G be a finite group acting linearly on a vector space V. We consider the linear symmetry groups $${\text {GL}}(Gv)$$ GL(Gv) of orbits $$Gv\subseteq V$$ Gv⊆V , where the linear symmetry group $${\text {GL}}(S)$$ GL(S) of a subset $$S\subseteq V$$ S⊆V is defined as the set of all linear maps of the linear span of S which permute S. We assume that V is the linear span of at least one orbit Gv. We define a set of generic points in V, which is Zariski open in V, and show that the groups $${\text {GL}}(Gv)$$ GL(Gv) for v generic are all isomorphic, and isomorphic to a subgroup of every symmetry group $${\text {GL}}(Gw)$$ GL(Gw) such that V is the linear span of Gw. If the underlying characteristic is zero, “isomorphic” can be replaced by “conjugate in $${\text {GL}}(V)$$ GL(V) .” Moreover, in the characteristic zero case, we show how the character of G on V determines this generic symmetry group. We apply our theory to classify all affine symmetry groups of vertex-transitive polytopes, thereby answering a question of Babai (Geom Dedicata 6(3):331–337, 1977. https://doi.org/10.1007/BF02429904).