Given a bounded lattice L with bounds 0 and 1, it is well known that the set Pol0,1(L) of all 0, 1-preserving polynomials of L forms a natural subclass of the set C(L) of aggregation functions on L. The main aim of this paper is to characterize all finite lattices L for which these two classes coincide, i.e. when the set C(L) is as small as possible. These lattices are shown to be completely determined by their tolerances, also several sufficient purely lattice-theoretical conditions are presented. In particular, all simple relatively complemented lattices or simple lattices for which the join (meet) of atoms (coatoms) is 1 (0) are of this kind.