The dynamical behaviour of an optimizing neural network is closely related to its parameters. For the transiently chaotic neural network (TCNN), the temperature, i.e., self-feedback weighting, is an important parameter for the network performance. While a high temperature is required to investigate chaotic dynamics, a low temperature is preferred for combinatorial optimization application. In this article, we derived this critical temperature of the TCNN analytically and illustrated its validity using computer simulation.