Nonnegative Matrix Factorization (NMF) has been a useful tool to solve the spectral unmixing for fluorescence imaging, which yields a set of constituent spectra, i.e., endmembers, and their corresponding fractional abundances. As observed from the spatial distribution of fluorescence data, target fluorophores are sparse and localized at certain regions while background fluorescence (including autofluorescence) are non-sparse and diffusive over large areas. Based on the different sparsity characteristics of abundances between target fluorophores and background fluorescence, we propose a NMF algorithm based on the target-to-background contrast with entire abundances being divided into a hierarchy of target fluorophores and a hierarchy of background. With the clear distinction between abundances of targets and background fluorescence in the iterative updates of NMF, appropriate sparseness constraint can be easily introduced into the corresponding target hierarchy without interfering with the other background hierarchy. Experimental results based on synthetic and real fluorescence data show the better performances of the proposed algorithm with respect to other state-of-the-art methods.