Previously, we proposed a novel classifier named kernel-based nonlinear discriminator (KND) to discriminate a pattern class from other classes. Since the solution is in a closed batch training form, it is inefficient to retrain a trained KND when novel data become available, or to obtain sparse representation for computationally intensive problems. This paper intends to solve the two problems by adopting an incremental learning procedure and a related feature reduction technique. Feasibility of the addressed methods is illustrated by experimental results on handwritten digit recognition.