Fractional calculus is defined by expanded integer order integration and differentiation. Most of research results describing mathematical modeling by fractional calculus deal with physical or electrical phenomena, not heat transfer phenomenon. Also, there is no research result that describes the effectiveness of operator based nonlinear controller with fractional calculus. In this paper, the mathematical modeling of heat transfer phenomenon of process control system by fractional calculus is proposed. Also, an operator based nonlinear control system with fractional calculus is designed. Then, we utilized a fractional PID controller as the tracking controller. Finally, simulation is conducted to verify the effectiveness of the proposed fractional controller. We confirm that the robust stability against the uncertainty of fractional dynamics is guaranteed.