This paper analyses the knowledge and software engineering aspects of educational applications that provide affective bi-modal human-computer interaction. For this purpose, a system that provides affective interaction based on evidence from two different modes has been developed. More specifically, the system’s inferences about students’ emotions are based on user input evidence from the keyboard and the microphone. Evidence from these two modes is combined by a user modelling component that incorporates user stereotypes as well as a multi criteria decision making theory. The mechanism that integrates the inferences from the two modes has been based on the results of two empirical studies that were conducted in the context of knowledge engineering of the system. The evaluation of the developed system showed significant improvements in the recognition of the emotional states of users.