Binocular rivalry occurs when a stimulus (e.g., horizontal bars) that is clearly visible, when presented to one eye, is periodically rendered invisible when a different stimulus is simultaneously presented to the other eye (vertical bars, for example). While the physiological basis of rivalry (and fusion) has been much debated, two general theories have been suggested. One theory argues that rivalry represents a competition between interpretations (or perceptions) that are independently generated by the different stimuli presented to the two eyes (which is comparable to viewing monocular ‘ambiguous’ figures). The second hypothesis suggests that the visual information from one eye is suppressed early in the visual pathway prior to the stage of binocular confluence (e.g. the lateral geniculate nucleus or layer IV of primary visual cortex). To test the validity of these and other possible explanations, subjects were first presented with two panels that elicit sensations of opposing saturated colors. When independently presented to the right and left eyes in dichoptic viewing, the images rival in perception. When, however, the right and left eye are presented with panels that have the same average chromaticity as in the previous experiment, but in a context that is more consistent with seeing the same object in the same location in space, the spectrally different stimuli fuse in perception for most observers. This effect is striking since extracting specific elements of the fused image, and presenting them independently on uniform chromatic surrounds causes the spectral elements to rival in perception. These results suggest that different spectra induce rivalry when the context of the stimuli are consistent with viewing different objects in the same location in space, but fuse when the information presented to the two eyes is consistent with viewing the same object in the same location. More generally, fusion and rivalry are not determined simply by the physical similarities and differences of the stimuli as such, but are manifestations of the coherent nature of the stimulus context.