Thermal modeling and optimal design of compact heat exchanger is presented in this paper. Fin pitch, fin height, cold stream flow length, no-flow length and hot stream flow length were considered as five design parameters. A CFD analysis coupled with artificial neural network was used to develop a relation between Colburn factor and Fanning friction factor for the triangle fin geometry with acceptable precision. Then, fast and elitist non-dominated sorting genetic algorithm (NSGA-II) was applied to obtain the maximum effectiveness and the minimum total pressure drop as two objective functions. The results of optimal designs were a set of multiple optimum solutions, called ‘Pareto-optimal solutions’. It reveals that any geometrical changes which decrease the pressure drop in the optimum situation, lead to a decrease in the effectiveness and vice versa. Finally sensitivity analysis shows the increases of heat transfer surface area necessarily do not increases the pressure drop and it is case sensitive.