Natural stimulus fMRI has been increasingly used in the brain imaging and brain mapping fields thanks to its more realistic stimulation of the brain's perceptive and cognitive systems. However, identifying consistent functional networks across different brains in natural stimulus fMRI data has been challenging due to the intrinsic variability of individual brain's responses and a variety of sources of noises. Inspired by recent promising results of sparse representation of whole-brain fMRI data, in this paper, we present a novel hybrid temporal and spatial sparse representation of whole-brain natural stimulus fMRI data for the inference of common functional networks across fMRI sessions and individual brains. Experimental results on natural stimulus fMRI dataset demonstrated the effectiveness of this framework.