Compact antenna test range (CATR) technology is key measurement technologies of antenna and target specific areas. Tri-reflector compact antenna test range is a promising solution of high frequency compact range systems. The system uses two shaped-reflectors as the feed system, making the quiet zone utilization significantly improved. A tri-reflector compact antenna test range system operating from 40GHz to 100GHz is designed and simulated in this paper. In order to reduce edge diffraction effect, the author proposes a series of measures to improve the performance. Simulation result in GRASP shows that within the quiet zone, the amplitude ripple is less than 1dB; phase ripple is less than 10 degree; x-polar isolation is greater than 30dB and the quiet zone usage is up to 70%. The system performance can meet system requirement very well. To provide guidance to system manufacture and calibration, much effort is put into system parameter studies and sensitivity analysis.