Recognizing abnormal breathing activity from body movement is a challenging task in machine vision. In this paper, we present a non-intrusive automatic video monitoring technique for detecting abnormal breathing activities and assisting in diagnosis of obstructive sleep apnoea. The proposed technique utilizes infrared video information and avoids imposing geometric or positional constraints on the patient. The technique also deals with fully or partially obscured patients' body. A continuously updated breathing activity template is built for distinguishing general body movement from breathing behavior