In this paper, we investigate how to appropriately use contextual information for object categorization tasks, especially in the statistical bag-of-features (BoF) framework. Our main contributions are two-folds. Firstly, we propose a context measuring mechanism which could explicitly assess roles of context features for different object recognition tasks. By analyzing information entropy and data ambiguity, only the useful and confident context information would have a final impact on categorization. Secondly, based on the context assessing results, under the BoF framework, we design unified object representations that incorporate the object appearance and contextual information from multiple spatial levels without the need of prior scene segmentations or context annotations. We evaluate the proposed method by the PASCAL object categorization task. The experimental results demonstrate that the proposed context modeling approach improves object categorization significantly and outperforms several state-of-the-art context models.