Handcrafted ordinal measures (OM) have been widely used in many computer vision problems. This paper presents a structured OM (SOM) method in a data driven way. SOM simultaneously learns ordinal filters and structured ordinal features. It leads to a structural distance metric for video-based face recognition. The SOM problem is posed as a non-convex integer program problem that includes two parts. The first part learns stable ordinal filters to project video data into a large-margin ordinal space. The second seeks self-correcting and discrete codes by balancing the projected data and a rank-one ordinal matrix in a structured low-rank way. Weakly-supervised and supervised structures are considered for the ordinal matrix. In addition, as a complement to hierarchical structures, deep feature representations are integrated into our method to enhance coding stability. An alternating minimization method is employed to handle the discrete and low-rank constraints, yielding high-quality codes that capture prior structures well. Experimental results on three commonly used face video databases show that our SOM method with a simple voting classifier can achieve state-of-the-art recognition rates using fewer features and samples.
