Energy consumption can be economized by using nanofluids as heat transfer medium in heat exchangers. Thermal performance of a helically baffled heat exchanger combined with a 3D fined tube operated with nanofluids is investigated numerically. Water based nanofluids of Cu, CuO, and CNT nanoparticles at different volume concentrations are considered. The effects of Reynolds number and volume concentration on heat transfer and pressure drop are evaluated. An increase in the volume concentration and Reynolds number intensified both heat transfer and pressure drop. For CuO/water and Cu/water nanofluid, the Nu Number increased as the volume concentration increased, while for CNT/water, the Nu number decreased as the volume concentration increased. Models of Nusselt number and pressure gradient are obtained in the heat exchanger in terms of Reynolds number, volume concentration and physical properties of particles by applying the neural network. The neural network predicts the output variables with a great accuracy.