This paper presents the hardware system scheme of multi-channel image processing based on DSP, with DSP as the core control of FPGA and CMOS, responsible for the overall scheduling system, using FPGA as coprocessor responsible for the collection of the original image information from CMOS. This paper combines the fisheye lens to obtain the complete original image for panoramic vision and stereo vision requirement. FPGA is used to design the SRAM cache controller based on Ping-pong mechanism. The data exchanging between DSP and FPGA are completed in EDMA model of DSP. Asynchronous FIFO is applied to solve the problem of cross clock domain. Experiments show that the system can be realized in the image data acquisition speed of 45 frames per second and the output of PAL style distortion fisheye image.