In this paper, we propose a novel method for robustly classifying visual concepts. In order to achieve this aim, we propose a scheme that relies on Self Organizing Maps (SOM [6]). Heterogeneous local signatures are first extracted from training images and projected into specialized SOM networks. The extracted signatures activate several neural maps producing activation histograms. These activation histograms are then combined into a global fusion process in order to build our final image representation. This fusion scheme is generic and shows promising results for automatic image classification and objectionable image filtering.