Real-time frame rate is an important factor for practical deployment of computer vision systems. Field programmable gate array (FPGA) technology has been considered for many applications due to its parallel computing capability. FPGA implementations of computer vision algorithms normally involve buffering data on external memory devices, which could slow down the whole system. This paper proposes a buffering scheme suitable for implementing real-time vision-based systems on an FPGA that does not require external memory to buffer data. A stop sign detection system implemented on an FPGA employing the proposed buffering scheme is presented as an example system. This system is capable of processing over 200 fps at the frame size of 480 × 752 pixels.