Binocular rivalry is observed when the stimuli presented to both eyes are incongruent and cannot be fused together into a single percept. In such a situation the brain perceives an alternation of the two images, with the mean duration of the dominant perception being a few seconds. We propose a computational model of competition with spiking frequency adaptation in the inferotemporal cortex (IT) that is able to reproduce the experimental distributions of the dominant phase durations. We also include top-down connections from IT to earlier visual areas (V4) that generate the observed perception-modulated response of some neurons in these areas.