Mathematical morphology is based on the principle of ordering. There is no natural way to order colors (being triplets of scalars). A lot of different ordering relations have been proposed in the literature, most on an ad-hoc basis. In this paper, we propose an ordering relation for colors that is based on the natural (i.e. physically plausible) ordering of spectra. Therefore, we ensure that the ordering is independent of the chosen color parameterization. We discuss that part of colorimetric theory that enables us to reconstruct the spectrum given the three color parameters. Furthermore, we present the very basics of the algebraic framework of mathematical morphology. This allows us to embed the presented ordering of colors within the morphological framework such that we can fully exploit the possibilities to define morphological image operators working on color images.