Consensus algorithm is widely used for Cooperative Spectrum Sensing (CSS) in Cognitive Radio Networks (CRNs). However, using this method, much of time and energy consumption for cooperation is wasted by repeatedly recirculating redundant information. We propose a novel Contextual Binary Gossip (CBG) algorithm for CSS. Utilizing the contextual information, two Secondary Users (SUs) will exchange sensing results only when they have different sensing or fussion decisions. We demonstrate that CBG has a faster convergence rate, thus more time can be used for SUs' data transmission. In addition, the energy consumption is reduced. We analyze the theoretical convergence time of CBG in three different network topologies respectively. For Cycle Graph and Grid Graph networks, CBG improves the convergence time of binary gossip by factors of n. For Random Graph network, CBG yields a n/ log n factor improvement. The simulation results validate the rapidity of our algorithm.