Formal Concept Analysis (FCA) is a principled way of automatically deriving ontology from larger textual data and interested by many researchers. Central of FCA is a pure and well defined formal context. Exploiting documents with characteristic words to form formal context, denoted as textual formal context (TFC), which lead to generate a highly sparse two-dimensional space with lots of noise. It seriously affects efficiency of constructing concept lattice as well as the structure of lattice. Therefore, it is necessary to find an effective method for reducing the textual formal context. Comprehensively considering the nature of textual formal context in this paper, we propose a method for reducing textual formal context on the view of semantic distance between attributes based on thesaurus. We propose two indicators to evaluate reduction of TFC, named as information losses entropy ILE and semantic coverage SC. Experimental results show that our proposed method is more competitive and promising.