Feature extraction is critical to the success of a face recognition system. Local Binary Patterns (LBP), with its different extensions, is one of the most popular texture descriptors, because of its demonstrated accuracy and efficiency. A LBP code is Jointly determined by a number of local comparisons between a central pixel and its surrounding pixels. Therefore even a single flipping of any comparison results will dramatically change the resulting LBP code. This paper proposes a novel feature descriptor, named Local Salient Patterns (LSP), which aims to only encode the most robust local comparisons, with the largest positive or negative contrast magnitude in LBP feature representation. Therefore LSP is expected to be more robust than the conventional LBP descriptor. In addition, LSP can be further extended to high order cases which explore more local relationships among multiple pixels. Extensive experimental results demonstrate that LSP outperforms the uniform LBP in most cases, when encoding using different radii and number of sampling points. LSP also achieves better performance than some advanced variants of LBP descriptors such as Local Ternary Patterns (LTP). We show that multi-order LSP achieves state-of-the art face recognition performance.