Whether a television (TV) program is good or not is generally judged by its TV rating, which indicates the audience size. However, it is obvious that a TV rating does not necessarily guarantee the quality of the program. In this chapter, we describe the architecture and experimental results of BROAFERENCE, which is an emotion-based TV quality rating system that makes it possible to judge the quality of a TV program by measuring emotional information from the TV audience. BROAFERENCE utilizes automatic detection of facial expressions and gaze information via a camera observing the TV program’s audience in order to collect emotional cues about the impact of a certain program as well as measuring the attention of viewers.