Topological mapping and scene recognition problems are still challenging, especially for online realtime vision-based applications. We develop a hierarchical probabilistic model to tackle them using color information. This work is stimulated by our previous work [1] which defined a lightweight descriptor using color and geometry information from segmented panoramic images. Our novel model uses a Dirichlet Process Mixture Model to combine color and geometry features which are extracted from omnidirectional images. The inference of the model is based on an approximation of conditional probabilities of observations given estimated models. It allows online inference of the mixture model in real-time (at 50Hz), which outperforms other existing approaches. A real experiment is carried out on a mobile robot equipped with an omnidirectional camera. The results show the competence against the state-of-art.