In this paper, we propose a delay constrained spatio-temporal video rate control method for lossy channels where the effective channel rate available to the video encoder is time-varying. Target bit-rate constraint for encoding is derived, which guarantees in-time delivery of video frames. By using empirically obtained rate-quantization and distortion-quantization relations of video, the distortions of skipped and coded frames in near future can be calculated in real-time. For the window expanding from the current to the firstly coded frame including the skipped frames in between, the number of frames to skip from the current and the quantization parameter(QP) for the firstly coded frame are decided in the direction to minimize the average distortion of frames in the window. From the simulation results, the proposed method is shown to enhance the average PSNR performance compared to TMN8 with some increase in the number of skipped frames and less number of delay violations.