This paper presents an unsupervised statistical model for simultaneous clustering, feature selection and outlier rejection in the case of count data. The proposed model is based on a finite discrete mixture to which a uniform component is added to ensure robustness to outliers and noise. The consideration of a finite mixture model is justified by its flexibility, its solid grounding in the theory of statistics and its competitive results. We derive a complete maximum a posteriori learning approach that does not require a priori knowledge about the number of outliers and the number of clusters. A rigorous expectation maximization (EM) algorithm, based on the formulation of a maximum a posteriori (MAP) estimation, is also provided. We report experimental results of applying our model to the challenging problems of visual scenes categorization and texture discrimination.