Background model can help to improve the compression efficiency for surveillance video coding, but the existing frame-based background model is inefficient in some situations, for example, when a region of background changes frequently or periodically. In this paper, a block-based background model is proposed to solve this problem. We save the background blocks recognized from each reconstructed frame into a buffer, thus the background blocks are collected gradually. At the same time, we compose a new background frame for each frame to be encoded based on the background blocks currently available in the buffer. Compared with the pre-built background frame, the instantly composed background frame often predicts more accurately because of the accumulated information about background. Experimental results show that the proposed model achieves better rate-distortion performance over the existing frame-based model in most cases, while keeping almost the same computation complexity.