This work presents a comparative analysis of Si/Ge and GaSb/InAs heterojunction Tunnel FET (TFET)-based cellular neural networks (CNNs). TFET-based CNNs are also compared against an equivalent FinFET-based CNN. A simulation methodology is shown to project realistic estimation of TFET-CNN performance based on the measured IDS-VGS characteristics of TFETs. III-V-TFET (i.e., GaSb/InAs TFET) shows a higher performance in CNN than Si/Ge-TFET due to a higher ON-current and much steeper switching slope (SS) in III-V-TFET. Meanwhile, Si/Ge-TFET shows a much lower OFF-current than III-V-TFET, and it is more suitable for ultralow power CNN applications. Cohesive simulation methodology discussed in the work also identifies that suppression of trap-assisted-tunneling (TAT)-induced leakage is critical to enable energy efficient TFET-based CNN. While a higher gate-to-drain capacitance (CGD, Miller capacitance) becomes a challenge in TFET-based digital designs, suitable design techniques are described to suppress its implications in throughput efficiency of TFET-CNN. Application of TFET-CNN is considered for image processing. Power-performance characteristics of CNN designs based on both the TFETs are compared.