With the development of widely-used unmanned aerial vehicles (UAV), automatic object recognition for UAV aerial images has important practical values. Since the background of objects is complex, there are limitations in object recognition using single-source visible or infrared data. Multi-source images contain much more information of objects, which can improve the recognition rate. Meanwhile there exist the problems of high dimension and nonlinear separability between features. In order to solve these problems, a recognition algorithm based on kernel dictionary learning is proposed. First, the algorithm learns a kernel dictionary and then obtains the sparse representations of objects by the kernel dictionary. Then the linear discriminant analysis is used to discriminate the sparse representations. Finally, the support vector machine is employed to classify four kinds of objects. The experimental results on visible and infrared images show that our method based on kernel dictionary learning has superior recognition performance in comparison with the methods based on traditional feature extraction and dictionary learning.